Emotion Detection Based on Consecutive Facial by Combining CNN and LSTM Implemented on FPGA Chip

1. Introduction

In recent years, with the continuous development of artificial intelligence (AI) technology, image and speech emotion recognition technologies have become one of the important applications. The AI technologies have significant application in many fields, such as driver monitor [1], fraud detection [2], medical care [3] and education [4]. In many application scenarios, embedded devices play an important role, so it has become particularly important to achieve neural network model inference on embedded devices such as FPGA chips.

For data with temporal dependencies, such as consecutive images, speech, and natural language, using temporal model can enhance the performance of the model. Therefore, the model architecture used in this paper first applies CNNs to extract local features from the data. To enhance the model’s learning performance on time-series level, LSTM neural networks are then used to model the feature sequences. Finally, FCNN is employed for results classification.

Emotion recognition is a technology that utilizes various signals, such as facial images and speech, to analyze and identify emotional states. In the case of using facial images for emotion recognition, the current main approach is to extract feature values from the image using 2-D convolutional neural networks, followed by prediction and classification, as demonstrated in papers [5,6]. Emotion recognition technologies are crucial for understanding and analyzing human emotions, as it can assist us in better understanding the emotional state of the user and provide more accurate and human-like solutions for related application areas [7]. Unlike speech recognition, where data preprocessing is crucial and involves steps such as audio cropping, noise reduction, and feature extraction, image recognition requires less preprocessing since raw images are typically used with minimal manipulation. Speech recognition heavily depends on high-quality preprocessing to handle variability, while image recognition can often learn features directly from raw images. Many studies have shown that the quality of data preprocessing and feature extraction significantly affects the performance of speech emotion recognition models. Experimental results in paper [8] show that the accuracy of machine learning models is affected by using different speech features for training. Consequently, in order to obtain accurate and stable performance of artificial intelligent models on emotion recognition, this paper focuses on the emotion recognition based on consecutive facial images.

However, the accuracies of facial recognition mainly depend on the quality of the input signals, such as the contrast, brightness, and focus of images for image signals. Consequently, it may significantly reduce the reliability of emotion recognition if any of the elements in the image fails to meet the standard requirements, such as overexposure or blurring. For the above reasons, this paper proposes consecutive facial pre-processing and recognition methods, to complete more suitable emotion recognition methods based on the user’s environment.

1.1. Field-Programmable Gate Array

The implementation of intelligent systems such as automatic emotion recognition technology in embedded systems faces many challenges, e.g., real-time requirements, resource constraints, and low power consumption requirements. Therefore, when implementing facial emotion recognition technology, the choice of hardware platform is crucial to the system’s efficiency and performance. Consequently, the FPGA (Field-Programmable Gate Arrays) have become one of the ideal platforms for implementing emotion recognition due to their highly customizable features, parallel processing capabilities, and low power consumption.

The FPGA is a reconfigurable embedded device commonly used in digital logic and digital signal processing applications. FPGA’s high flexibility and programmability enable their wide application in various fields, including IC testing [9], embedded systems [10,11,12], and the IoT (Internet of Things) [13].

The features of FPGA include:

• Reconfigurability: FPGAs are reconfigurable [14] and can define their digital logic circuits through programming, allowing developers to redesign the FPGA’s functions according to application requirements repeatedly.
• High parallel processing capability: FPGAs have multiple independent logic circuits and data paths that can run in parallel, enabling them to efficiently perform parallel processing for multiple tasks and hence provide high-performance computing power.
• Low latency and high-frequency operation: Due to the fact that FPGA’s logic circuits composed of gate arrays and have high optimization capabilities, it can achieve low latency and high-frequency operation. This makes it ideal for applications requiring high-speed processing.
• Customizability: FPGAs are highly flexible in customization and can be designed and optimized according to application requirements. This includes design of logic circuits, data paths, memory, and interfaces.
• Software and hardware co-design: FPGAs provide the ability to co-design software and hardware on a single chip [15]. This provides higher system integration and performance.
• Suitable for rapid development and testing: FPGAs have a rapid development cycle. Developers can quickly develop and test them within a shorter period [16].

1.2. Experimental Protocol

This paper utilizes two deep learning frameworks, TensorFlow and Keras, to train emotion recognition models for consecutive facial images signals on a PC. The parameters of the models are transferred to an FPGA chip. The neural network model inference algorithms are used to simulate computation of model inference in the deep learning frameworks and then obtain the final classification results. For consecutive facial emotion recognition, this paper dynamically captures 30 frames of images from a video as the consecutive image data. The facial images are extracted by using the open-source face detection model from OpenCV. The CLDNN (Convolutional Long Short-Term Memory Fully Connected Deep Neural Networks) model architecture proposed in paper [17] is used to build and train the ML model. The trained model is then deployed on the FPGA chip for model inference.

2. Related Works

In this section, the CLDNN model [17] is introduced. Moreover, some related researches of consecutive facial emotion recognition, and the implementation of model inference on FPGA are also explored in this section.

2.1. CLDNN Model Architecture

The paper [17] proposes a neural network model architecture called Convolutional Long Short-Term Memory Fully Connected Deep Neural Networks, referred to as CLDNN or ConvLSTM-FCN, for processing time series data such as audio signals, consecutive images, and natural languages. The CLDNN model is composed of convolutional neural networks (CNNs), LSTM neural networks, and fully connected neural networks of DNNs. The CNNs and LSTM neural networks analyze and calculate local features and sequential patterns in the data, respectively. The fully connected layers are then classifying the prediction results. Since consecutive images have time-series characteristics, they are suitable for training with recurrent neural networks (RNNs) such as LSTM. According to [17], the CLDNN model outperforms those using only CNN or LSTM models in terms of the accuracies of the models.

Therefore, this paper applies and modifies the CLDNN model architecture for emotion recognition, combining CNNs, LSTM neural networks, and DNNs to build the ML model. The Figure 1 shows the architecture of the CLDNN model.

2.2. Consecutive Facial Emotion Recognition

Paper [18] proposes a method for consecutive facial emotion recognition. Firstly, the frame interval is calculated based on the average duration of all video files in the database. Frames are then extracted base on this interval to obtain a segment of consecutive image data. Next, facial landmark detection (68 points) [19] is used to identify the positions of the facial features in the images, and the Euclidean distances between these points are used as the feature value. The image feature values of the same image of consecutive image data are concatenated into a feature sequences. Finally, these feature sequences are used as inputs to train an LSTM model. This paper achieves an average accuracy of 98.9%.

A multimodal emotion recognition model that combines video data and audio data is proposed in [20]. For the emotion recognition of consecutive facial image, the authors randomly extract 30 images from video data. Then, using the MTCNN (multi-task cascaded convolutional network) proposed in paper [21], they extract the facial parts from all images in each image. The images are then reshaped uniformly to (160, 160) color images as data features. Hence, the input of the facial emotion recognition model is a data sequence of shape (N, 30, 160, 160, 3), where N is the number of video data. In the speech emotion recognition model, the log Mel-spectrogram is extracted from every segment of the speech signals and used as input data of ML model. In order to obtain the log Mel-spectrogram, the authors utilize 94 Mel-filter banks for each speech signal, with a 40ms Hanning window and 10ms overlapping. This result in a representation with dimensions of 94. After performing cross-validation with the ResNet-50 model, the paper find that the consecutive facial emotion recognition model achieves an accuracy of 95.49% and the speech emotion recognition model achieves an accuracy of 75.61%. The fusion model combining consecutive facial image and audio achieves an accuracy of 97.57%.

In paper [22], the authors propose a method that uses facial landmarks [19] to detect facial regions in images. The detected facial regions are then converted into grayscale images. The authors extract 32 features from the resulting images using Gabor filters. The features are then combined with the 68 positions of the facial landmarks. After processing all frames in the video, the authors obtain 2176 (32×68) features for each facial image. The proposed method in [22] achieves an accuracy of 96.53% on testing. Paper [23] proposes a multimodal emotion recognition model that combines consecutive facial image and speech. Regarding the consecutive facial emotion recognition model, the authors use the InceptionV3 model to extract feature values from single images, which are then sent to LSTM for time-series learning. This results in an accuracy of 94% on RAVDESS database. In terms of speech emotion recognition, the authors extract MFCCs from the speech as feature values and train them using the CLDNN model. The experimental results show that the speech emotion recognition model proposed in [23] achieves an accuracy of 82%. The fusion model combining consecutive facial image and audio achieves an accuracy of 96%.

3. Facial Emotion Recognition Methods and Parameter Setting

This section will introduce the training and testing process of the proposed consecutive facial emotion recognition method in this paper. Some experiments and the experimental environment will be also introduced in this section.

3.1. CLDNN Model

When conducting experiments using the RAVDESS database, this paper tests different model performances by employing both LSTM and GRU as temporal neural networks. The experimental results demonstrate that in the consecutive facial emotion recognition model, using LSTM to process temporal data achieves an accuracy of 99.51%, showing a 1.65% improvement compared to using GRU. As LSTM neural networks yield higher accuracy in consecutive facial emotion recognition models, and using an LSTM model for model inference on an FPGA chip does not show a significant increase in inference time compared to using a GRU model, this paper opts to use LSTM as the modeling approach.

Table 1. The difference in model accuracy and execution time on an FPGA chip between models built using LSTM and GRU.

CNN + LSTM + DNN		CNN + GRU + DNN
Execution Time on FPGA	Accuracy	Execution Time on FPGA	Accuracy
11.70 sec	99.51%	11.67 sec	97.86%

Table 1. The difference in model accuracy and execution time on an FPGA chip between models built using LSTM and GRU.

CNN + LSTM + DNN		CNN + GRU + DNN
Execution Time on FPGA	Accuracy	Execution Time on FPGA	Accuracy
11.70 sec	99.51%	11.67 sec	97.86%

3.2. Experimental Environment for Model Training on PC

This paper performs data preprocessing on the CPU of a PC, and trains a CLDNN model for consecutive facial emotion recognition using GPU. The trained parameters for CLDNN model are transferred to an FPGA chip for inference of the model. Table 2 shows the hardware specifications and software environment used for training the model on the PC.

The experiments conducted in this paper uses 10-fold cross-validation method for training and testing. In order to ensure the reliability of machine learning model, the Scikit-learn package is conducted to randomly select 10% of the dataset as the data for testing, and the remaining 90% data are randomly divided into train data and validation data at a ratio of 9:1. Figure 2 illustrates the data proportions of the training set, validation set, and testing set.

3.3. Consecutive Facial Emotion Recognition

In this subsection, a detailed description of the databases, data preprocessing methods, neural network model architecture, and parameter settings for proposed consecutive facial emotion recognition method will be provided. Then, the experimental results of the proposed consecutive facial emotion recognition method will be compared with other related literature.

3.3.1. Databases

This paper uses the RAVDESS [24], BAUM-1s and eNTERFACE’05 databases for training and testing the proposed consecutive facial emotion recognition model.

3.3.1.1. RAVDESS

RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) is a database of consecutive emotional image and speech created by Ryerson University in Canada. It includes emotional expressions in both visual and auditory forms from different actors and can be used for research in the fields of emotion recognition, emotion analysis, audio processing, etc.

This database contains videos and voice data from 24 actors (12 males and 12 females) from Canada. Each actor records videos and audio in 8 different emotional states (Angry, calm, disgust, fear, happy, neutral, sad, and surprised), by speaking and singing. For each emotional state, the actor performs multiple times to provide data with different emotional intensities and expressions. The RAVDESS database has a total of 2,452 consecutive image and audio data, with the distribution and the amount of data for each emotional state shown in the Table 3.

3.3.1.2. BAUM-1s

The BAUM-1s database was created by the Department of Electrical and Electronics Engineering at Bahcesehir University in Turkey. This database consists of samples of videos and voice recordings, recorded by 31 professional actors. In this study, we utilize six emotional categories: angry, disgust, fear, happy, sad, and surprised, to conduct experiments of consecutive facial and speech emotion recognition. The data number and proportion of each emotion label in the BAUM-1s database is shown in Table 4.

3.3.1.3. eNTERFACE’05

The eNTERFACE’05 database was recorded by 44 professional actors and consists of six different emotion categories, with different sentences expressing each category. This database includes both videos and speech samples, making it commonly used in various multimodal emotion recognition research. The data number and proportion of each emotion label in the eNTERFACE’05 database is shown in Table 5.

3.3.2. Pre-Processing

This paper refers to [18] and adopts a dynamic sampling method to capture 30 frames of video file to represent a consecutive image data. The frame intervals are determined by dividing the total number of frames in each video data by 30. Then the frame intervals are used as a spacing to capture 30 frames in each video. The Figure 3 shows the process of the dynamic capture of the consecutive images.

The OpenCV facial detection model is then used to locate the coordinates of the faces in the images, and any unnecessary data outside the coordinates is discarded. In order to reduce the computational load during model inference on the FPGA chip, the color images are converted into grayscale images. Moreover, all facial images are resized to a uniform size of 100×100 pixels. The Figure 4 shows the pre-processing flowchart for the facial emotion recognition method proposed in this paper.

3.3.2.1. Facial Detection

Facial detection is necessary before training facial emotion recognition models. Nowadays, there are many open-source facial detection models available, such as facial landmark detection and the MTCNN model proposed in [20]. In this paper, Haar cascade frontal face detection model provided by OpenCV is used, which is also used for facial detection in many related researches, such as the papers [25,26]. This facial detection model can locate the starting coordinates (x, y) of the facial region in the images, and the corresponding w (width) and h (height). The facial image within the region is then extracted, while unnecessary information outside of the region are discarded. The process of the facial image extraction is shown in the Figure 5.

3.3.2.2. Grayscale Conversion

Paper [27] compares the performance of color and grayscale images in facial recognition. It finds that in some cases, using grayscale images performs facial recognition better than using color images. Moreover, using color images increases the computations and parameters during recognition. Overall, there are advantages and disadvantages to using color or grayscale images for facial recognition. Considering the restriction in computational requirements of performing neural network model inference on embedded devices, this paper converts facial images to grayscale as the inputs to the neural network model.

3.3.2.3. Resize

As the OpenCV face detection model is used to extract facial images, the sizes of facial images are different. Therefore, all facial images need to be resized into a unified size of 100×100 pixels before being fed into the neural network model for training and testing.

3.3.3. Experiments

This paper proposes a consecutive facial emotion recognition method based on the CLDNN model architecture. The LFLBs (Local Feature Learning Blocks) with a CNN in the neural networks is used as the main body to extract local features of input facial images. Every 30 of image features are concatenated into consecutive image feature sequences. Then, the LSTM layer is applied to strengthen the learning of model on time-series. Finally, a fully connected layer is used for classification to output the recognition result.

3.3.3.1. Local Feature Learning Block

The LFLB is a type of neural network module used in deep learning, which is primarily used to extract local feature values from input data. LFLB consists of many sub-modules, such as convolutional layers, pooling layers, and fully connected layers, etc. It can extract features from different regions of the data and share weights to reduce the number of model parameters and improve the model’s generalization ability. Some researches of facial emotion recognition also use LFLBs to extract local features from image data, such as the papers [28,29].

In this paper, a 2-D convolutional layer, a batch normalization layer, and a max pooling layer are used as the LFLB for extracting local features from the facial image data. Due to the local perception characteristics of the CNNs, it is suitable for calculating the local features of the image data and generating feature maps. The batch normalization layer can normalize the input data, scales the values to a specific range 0 ~ 1. It helps the model train the data faster and reduces the possibility of overfitting. The max pooling layer is mainly used to limit the sizes of the feature map. It helps preserve the most influential feature values (maximum values). This paper uses the zero-padding method in the design of sub-modules of the LFLB to ensure that the size of the feature map is the same as that of the input data, and to avoid data loss at the edges. Moreover, ReLU (rectified linear units) is used as the non-linear activation function of the LFLB. The Figure 6 shows the schematic diagram of the LFLB applied in this paper.

3.3.3.2. Training Process and Parameters

When conducting experiments using the RAVDESS database, setting the number of local feature learning blocks to 6 results in 64 local features per image and achieves the highest accuracy of 99.51%. However, from Table 6, it can be seen that when increasing the number of local feature learning blocks to 7, the output of local features per image decrease to 16, leading to a sharp decline in accuracy, reaching only 32.68%. Therefore, this study sets the number of local feature learning blocks in the consecutive facial emotion recognition model to 6.

The proposed strategy in this paper for consecutive facial emotion recognition is then described as follows. Firstly, local features will be extracted from single facial images using LFLB, and then concatenates the resulting 30 feature maps into a sequence. The sequence represents the features sequences of consecutive facial images. Then, the sequences are flattened and reshaped into the shape of [time step, data length], before being fed into the LSTM layer. Next, the output data are normalized after learning on time-series level in the LSTM layer. Finally, softmax method is performed when using a fully connected layer to extract the emotion label corresponding to the maximum output value for the sequence as the predicted result. The training process of the proposed consecutive facial emotion recognition method is shown in Figure 7.

The number of memory units in LSTM layers can affect the training performance of machine learning models and needs to be adjusted according to the type of data and the length of the data sequences. In this paper, for the consecutive facial recognition CLDNN model, cross-validation is used to test the effect of different numbers of memory units in the LSTM layer for finding the best model’s accuracy. The number of memory units is set up from 5 to 50, with an increment of 5. For applying 10-fold cross-validation method, the experimental results show that the highest accuracy of 99.51% is achieved when the number of memory units in the LSTM layer is 20. Therefore, this paper sets the memory units of the LSTM layer in the CLDNN model for consecutive facial emotion recognition to be 20. Figure 8 shows the experimental results for the relationship between the number of memory units in the LSTM layer and the accuracy of the proposed consecutive facial emotion recognition model. The proposed model’s parameters are listed in the Table 7.

4. Experimental Results

In this section, the experimental results on RADESS, BAUM-1s and IEMOCAP database will be shown and discussed. Then, some comparison between the results of consecutive facial emotion recognition method proposed in this paper with those of other related researches will also be made in this section.

4.1. Experiments on RAVDESS Database

For RAVDESS database, the pre-processing and emotion recognition methods in Section 3 are applied for this experiments. According to the experiments, the proposed consecutive facial emotion recognition method achieves an average accuracy of 99.51% on testing through 10-fold cross-validation experiments. The experimental results for the loss and accuracy of each fold of training, validation, and testing are shown in the Table 8.

Besides, the normalized confusion matrix obtained from cross-validation is shown in Figure 9. The accuracy, precision, recall, and F1-score for every emotion label can be calculated using equations from (1) to (4) with the confusion matrix in Figure 9, and the results are presented in Table 9.

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(1)

$$Precision=\frac{TP}{TP+FP}$$

(2)

$$Recall=\frac{TP}{TP+FN}$$

(3)

$$F1-score=\frac{2*Precision\ast Recall}{Precision+Recall}$$

(4)

where $TP$, $FP$, $TN$ and $FN$ is the number of $TruePositive$, $FalsePositive$, $TrueNegative$ and $FalseNegative$, respectively.

This paper proposes a consecutive facial emotion recognition method and conducts experiments on the RAVDESS database. The experimental results in Table 9 show that this method performs well in terms of accuracy, precision, recall and F1-score, only performs a little poorly in predicting the emotion label of “neutral”. This is because that the facial features of the emotion label “calm” are similar to those of the emotion label of “neutral”. The degree of expression variation between these two emotions is small. This makes it more likely to be predicted incorrectly. However, our proposed method achieves an overall performance of average value over 99% in the accuracy, precision, recall and F1-score on the RAVDESS database. This verify the proposed model is precise and reliable. Based on these results, the method proposed in this paper is a feasible emotion recognition method and will have a stable performance in practical applications.

Table 10 presents a comparison of the experimental results of the proposed consecutive facial emotion recognition method with other related researches on RAVDESS database. It can be seen that the proposed method achieves an accuracy rate of 99.51%, which is higher than the methods proposed in other related researches.

4.2. Experiments on BAUM-1s Database

Due to the data imbalance in the BAUM-1s database in which the emotion label “fear” accounts for only 6.99% of the data, and the emotion label “surprised” accounts for only 7.90%, the lower amount of learning data for these emotion labels during model training will affect the overall accuracy of the model. Therefore, this paper utilizes up-sampling to balance the data within the training dataset by replicating data instances from the minority classes, thus ensuring an equal number of data samples for all emotion categories. Figure 10 shows the distribution of the BAUM-1s database before and after up-sampling.

Applying the pre-processing and emotion recognition methods in Section 3 for this experiments, the proposed consecutive facial emotion recognition method in Figure 7 achieves an average accuracy of 87.80% on BAUM-1s database through 10-fold cross-validation experiments. The experimental results for the loss and accuracy of each fold of training, validation, and testing are shown in the Table 11.

Besides, the original and normalized confusion matrix obtained from cross-validation experiment on BAUM-1s database is shown in Figure 11 and the accuracy, precision, recall, and F1-score for each emotion are as shown in Table 12. Moreover, Table 13 shows the comparison of the accuracy of the proposed method for consecutive facial emotion recognition and those of other related researches for BAUM-1s database. Again, according to the comparison in Table 13, the proposed strategy for consecutive facial emotion recognition outperforms the methods in the other papers.

4.3. Experiments on eNTERFACE’05 Database

Similar to the experiments conducted in Section 4.1 and Section 4.2 for the databases BAUM-1s database and RAVDESS database, respectively, the proposed consecutive facial emotion recognition method in Figure 7 achieves an average accuracy of 96.82% for the experiments on eNTERFACE’05 database through 10-fold cross-validation method. The experimental results for loss and accuracy of each fold of training, validation, and testing are shown in the Table 14.

Besides, the original and normalized confusion matrix obtained from cross-validation experiment on eNTERFACE’05 database is shown in Figure 12 and the accuracy, precision, recall, and F1-score for each emotion are as shown in Table 15. Table 16 shows the comparison of the accuracy of the proposed method for consecutive facial emotion recognition and those of other related researches for eNTERFACE’05 database. According to Table 16, it is obvious that the proposed methods achieves a much higher recognition rate than those by using the methods in the other papers. This again verifies the performance of the proposed method.

5. Experiments for FPGA Implementation

This paper implemented the proposed consecutive facial emotion recognition pre-processing methods and deep learning model introduced in Section 3 and Section 4 on FPGA chip and then tested the performance of the chip using 10-fold cross-validation. Each testing dataset consisted of 246 consecutive image data. There are 30 facial images in each consecutive image data. The FPGA in DE-10 Standard development board is equipped with a dual-core ARM Cortex-A9 processor. In order to accelerate computation of the model on FPGA, the test dataset is divided into two sub-datasets, and parallel computing was used to implement the neural network model inference. This reduces the execution time by half. The accuracy of each fold in the test and the average test time for a single consecutive image are shown in Table 17.

The experimental results show that the testing accuracies on the FPGA development board using the proposed neural network model inference algorithms are the same as those tested on PC using deep learning frameworks such as TensorFlow and Keras for implementing the proposed model. This verifies that the implementation of the proposed neural network model on FPGA chip using inference algorithms in this paper is efficient and performs well according to Table 17. Moreover, Table 18 shows the average execution time and proportion of each convolutional layer, batch normalization layer, max pooling layer, LSTM layer, and fully connected layer when testing a single consecutive image data for more detailed analysis. It can be seen that due to the first convolutional layer expanding the dimensions of the image data to 32, the input data for the second convolutional layer is much larger than that of the first layer. As a result, the execution time of the second convolutional layer is the longest at 6.9281 seconds (59.21%). Furthermore, since the proposed consecutive facial emotion recognition model incorporates max pooling layers, which can reduce the dimension of the feature maps, the execution time of each convolutional layer decreases sequentially. However, the average execution time for running the proposed deep learning model CLDNN on recognize the emotion from a video costs only 11.7 seconds. This verifies that the implemented AI (Artificial Intelligent) chip based on FPGA is feasible and is suitable for the AI edge computing application.

6. Conclusions

This paper proposed the methods based on deep learning for consecutive facial emotion recognition. The proposed model was implemented on an embedded system with FPGA chip without the need for a deep learning framework during the model inference process. For consecutive facial emotion recognition, this paper captured 30 frames of an image sequence to represent a consecutive image segment. The Haar cascade frontal face detection model from OpenCV was utilized to extract the facial regions from the images, followed by grayscale conversion and resizing to reduce computational burden on the embedded device. The preprocessed images were then fed into local feature learning blocks to extract local features from individual frames. These features were then packaged into a feature sequence representing a consecutive image segment. The feature sequence was then passed through an LSTM layer for temporal sequence learning. Finally, a fully connected layer was used for classification.

Next, the parameters of the deep learning models for consecutive facial emotion recognition, as well as the test dataset, were fed into FPGA’s memory for model inference. This research implemented the neural network model inference algorithms in Python. Then, through high-level synthesis, the algorithms were automatically transformed from the high-level language into circuit functionality. This allowed us to realize model inference on the embedded device without the need of deep learning frameworks. For the model inference of consecutive facial emotion recognition, the proposed method achieved the same test accuracy as that tested on a PC using deep learning frameworks. This indicated that the neural network model inference algorithms proposed in this paper can achieve the same performance as using the deep learning frameworks. The average testing time for a single consecutive image data was 11.70 seconds, with an average testing time of 0.39 seconds per single image of size 100×100 pixels. The implemented hardware had an FPS of 2.56. The experimental results for the designed FPGA chip verify that the implemented AI (Artificial Intelligent) chip based on FPGA is feasible and is suitable for the AI edge computing application.

Finally, according to the experimental results in Section 4, the proposed deep learning model applied on the three databases RADESS, BAUM-1s and IEMOCAP databases achieves much higher recognition rates than those in the other papers. This demonstrate that the proposed methods outperform the methods in the other literatures.