EEG-Visual Multiclass Classification Based on a Channel Selection, MNE Algorithm and Deep Network Architectures

1. Introduction

Brain-computer interfaces (BCI) based on electroencephalographic (EEG) signals are gaining considerable expectation in the areas of scientific research and application development [1], essentially because of technological advances and multidisciplinary studies on brain signals [2,3]. Typically, depending on the category of EEG signals to be processed, countless BCI systems have been developed for ordinary use and clinical applications. This is the case for brain-controlled vehicles [4], drones[5], assistive devices [6], intelligent systems [7], neurorehabilitation [8], telemedicine [9], assistive robots [10], and wheelchairs [11], to name but a few. In several of these BCI applications, disposing of EEG signals with features being tagged in class labels is an advantage in controlling several tasks, according to the BCI system functioning logic. A complex BCI system needs more class labels than a classic one. In this sense, controlling an assistive robotic arm using motor imagery EEG signals, Onose et al. [12] needed four mental tasks randomly distributed in sequences of 35 trials each, while for their part, Zhu et al. [13] controlled a robotic arm with six degrees of freedom using fifteen target classes. Statistically, the ranking metric uses the probability that an input data stream belongs to one of the implemented model’s output labels. The more labels there are to classify, the less likely a data sequence will be correctly classified. Therefore, the challenge of classifying EEG signals into multiclass arises to meet this benefit of complex BCI systems. However, increasing tremendously the number of classes challenges the computational task of properly classifying a new feature vector into one of them, as mentioned by Del Moral et al. [14]. Besides, EEG signals naturally carry their own processing inherent challenges, as they are produced by non-stationary mental states [15]. The non-stationarity and non-linearity characteristics of EEG signals impose a rigorous and adequate use of processing algorithms for BCI systems that involve an outstanding level of precision for successful applications [16].

To address the challenge of accurately classifying EEG signals into multiclass labels for BCI systems, various algorithms based on robust model architectures have been proposed in recent literature, taking advantage of the availability of public databases. Mahmood et al. [17] proposed a multiclass classification algorithm based on Common Spatial Pattern (CSP) and Support Vector Machine (SVM) for BCI applications. They achieved an average accuracy of 85.5% using four frequency bands to classify motor imagery EEG signals into four classes. Recently, to be classified into five classes, motor imagery EEG signals were processed as 3-channel images using Deep Convolutional Neural Networks (DCNN) and Long Short Term Memory (LSTM) networks [18]. The model achieved an average accuracy of 70.64% using the Physionet dataset for EEG motor imagery tasks. Another innovative approach to improve BCI performance in multiclass classification consists of combining two different BCI modalities. In this sense, Kwon et al. [19] implemented a compact hybrid BCI based on EEG and fNIRS using channel selection and source-detector (SD) pairs approaches, respectively. A high classification accuracy of 77.6% was obtained by classifying three mental states. For their part, addressing the multiclass classification challenge for BCI systems, Spampinato et al. [20] released a database of EEG signals captured after six subjects visualized image sequences. Initially, using Recurrent and Convolutional Neural Networks (RNN, CNN) to learn and classify visual stimuli–evoked EEG from 40 ImageNet object classes, an average accuracy of 82.9% was achieved. Next, focusing on decoding visual information from brain signals with the same database and a multi-modal approach based on joint learning, Palazzo et al. reported accuracies of 60.4, 50.3, 54.7, 46.2, and 60.4% employing the Inception-V3, ResNet-101, DenseNet-161, AlexNet, and EEG-ChannelNet networks, respectively.

To address the challenge of EEG visual multiclass classification, the present work hypothesizes the suitable use of a reduced number of channels and two deep learning networks. Concretely, a mutual information method based on cross-entropy was developed, allowing the grouping of discriminant channels. Once the set of 54 discriminant channels is constituted, the preprocessing step is carried out by the minimum-norm estimates algorithm (MNE-Python) [21,22]. Regarding the classification step, the proposed method uses two classifiers. The first approach implements the EEGNet network whose excellent performance has been proven in the literature because of its temporal and spatial filter banks [23,24]. The second uses a combination of CNN and LSTM networks to extract and classify EEG features into 40 targeted classes, respectively. Therefore, the PL dataset published in [20] and the k-fold cross-validation technique are used to assess the proposed method. Outstandingly, the paper’s contributions are summarized as follows.

• A visual multiclass classification approach based on a reduced number of EEG channels and deep learning architectures is proposed for BCI applications.
• Multiclass classification comparative results using EEGNet and CNN-LSTM networks are provided.
• A channel selection approach based on mutual information is implemented to accurately discriminate contributing channels.

The results achieved in this paper offer new possibilities for multi-task embedded BCI applications based on EEG signals. The paper is organized as follows. Section 2 presents selected works related to EEG visual classification, emphasizing those where the dataset proposed in [20] is used. The method developed in this work is explained in Section 3, contemplating the high-level general diagram, the dataset employed, signals enhancement, and processing techniques. Finally, the results obtained are reported and discussed in Section 4 while Section 5 gives the paper’s conclusion and projects future works.

2. Related Work

Nowadays, research focused on modeling complex cognitive events from EEG signals is still attracting great interest due to the flourishing deployment of BCI systems. As an assistive device, BCI systems based on EEG signals are designed to meet the user’s commodity and adaptability. Based on a user’s mental stimulation, event-related potentials (ERP) are generated by an external inducement like a cognitive load, or auditory, sensory, or visual stimuli. Also, ERP is present in internal events such as stress, directed thought, or memory concentration [25]. Particularly, Visual-evoked potentials (VEP) considered in this work are sensory-evoked potentials induced by visual stimuli [26]. In this sense, Yedukondalu and Sharma [27] implemented K-nearest neighbors (KNN) and support vector machine (SVM) classifiers to identify cognitive load during mental arithmetic tasks, achieving an accuracy of 96.88%. In parallel, EEG signals from six auditory stimuli were classified for BCI applications utilizing classifiers based on random forest, multi-layer perceptron, and decision tree architectures [28]. Average accuracies of 91.56%, 89.92%, and 86.78% were reported, respectively. For their part, Kalafatovich et al. implemented a two-stream convolutional neural network to classify single-trial EEG signals evoked by visual stimuli into two and six semantic categories [29]. They achieved accuracies of 54.28±7.89% for 6 classes and 84.40±8.03% in the case of 2 classes. A short time ago, EEG signals induced by visual stimuli evoked by 40 image classes of the ImageNet dataset were classified using RNN and CNN [20]. A maximum accuracy of 82.9% was achieved by classifying EEG signals corresponding to image sequences. That work addressed the recent challenge of EEG multi-class classification, by offering a reliable alternative for multi-task BCI-based applications. Indeed, for BCI-based robotic applications, for example, a BCI system based on multi-tasks allows covering the robot’s degree of freedom in the sense that each mental task controls the robot’s considered movement. Therefore, taking advantage of the PL dataset availability, Xiao Zheng and Wanzhong Chen proposed attention-based Bi-LSTM models to classify into 40 classes, EEG signals evoked by image visualization [30]. Among other results, they reported classification accuracies of 96.27% and 99.50% using 29 prefrontal and occipital channels, and all 128 channels of the database, respectively. On the other hand, they partitioned the EEG sequence of 500 ms in Visualization-time segments of 40–200, 40–360, and 360–480 ms to evaluate classification accuracy improvement. The PL dataset was also used in [31], where a combination of ensemble and deep learning models allowed extracting category-dependent representations of EEG signals. The proposed LSTMS_B model achieved an average accuracy of 97.13% classifying EEG visual segments into 40 classes. Another recent work using the PL dataset was proposed by Kumari et al. [32]. EEG signals evoked from visual stimuli were processed as spectrogram images using a Capsule Network (EEGCapsNet) based on the Short-Term Fourier Transform (STFT). An average accuracy of 81.59% was reported, classifying EEG representations into 40 classes. Finally, a functional connectivity-based geometric deep network (FC-GDN) was proposed by Nastaran et al. to classify into 40 classes, EEG recordings evoked by images [33]. They obtained an average accuracy of 98.4% classifying EEG signals provided by the PL dataset. Table 1 presents related works in the recent literature, where the PL dataset has been used.

3. Methods

The proposed method focuses on accurate EEG signal processing, emphasizing the mutual information (MutIn) approach to select discriminant channels and the MNE algorithm to enhance signals. Next, EEGNet and CNN-LSTM classifiers are used to estimate the output class label probability. In other words, this work aims to classify EEG signals from a reduced number of channels, while providing reliable classification accuracy. Because the capture system uses 128 electrodes, organized according to the 10-20 system [34], and considering the brain cortex-specific functions [35], selecting discriminant channels allows constituting a channel subset that optimizes the classifier performance. Once the discriminant channel subset is constituted, EEG data are preprocessed by using the MNE-Python package [21], essentially for the time–frequency analysis. This signals analysis aims to efficiently make the data more suitable for classifiers by inspecting artifacts in both time windows and frequency patterns. In the last step, EEGNet and CNN-LSTM networks are used separately to extract and classify feature data.

3.1. Overall Flowchart

Figure 1 presents the high-level general diagram of the proposed method. Five gradual steps are contemplated in the method development. The first consists of downloading and preparing data from the referenced dataset. In the second stage, a channel selection approach based on MutIn is developed to build a subset of 64 discriminating channels [36,37]. Next, the Minimum-norm estimates algorithm (MNE) implemented in Python language is used to preprocess data from selected channels. Finally, the EEGNet and CNN-LSTM models classify EEG segments into 40 classes, separately providing the output.

3.2. The Perceive Lab Dataset

The dataset used in this work is provided by Spampinato et al. [20]. 2000 classic images from the Imagenet dataset [38] representing 40 labels of 50 pictures each, served as visual stimuli to six test subjects. The Imagenet dataset provides 40 image labels containing cats, sorrels, elephants, fish, dogs, airliners, brooms, pandas, canoes, phones, mugs, convertibles, computers, fungi, locomotives, espresso, chairs, butterflies, golf, piano, iron, daisy, jacks, mailbags, capuchin, missiles, mittens, bikes, tents, pajama, parachutes, pools, radios, cameras, guitar, guns, shoes, bananas, pizzas, and watches.

The experiment protocol consisted of instantly visualizing in a continuous sequence, blocks containing images of each label. The display duration of each image block was 0.5 seconds, followed by 10 seconds of recording EEG signals corresponding to the previous visualized block, before considering the same cycle for the adjacent image block. Each image block segment contains EEG signals from 128 channels, recorded with Brainvision DAQs equipment at the sampling frequency of 1 kHz. The data resolution was set to 16 bits. The electrode map of the used system, depicted in Figure 2, is based on the 10–20 placement system. Therefore, Equation 1 presents the number of EEG signal samples corresponding to each channel.

$$N_{samples}=\left(0.5s\right)*1kHz=500$$

(1)

All channels’ cues were intentionally scratched into 440 samples per task to avoid undesirable signals related to potential interference between the previous and recent image blocks, and also for the uniformization of pattern length purposes. Three band-pass filters were applied to the signals constituting the dataset. That is precisely a second-order Butterworth filter from 5 — 95 Hz, another from 14 — 70 Hz, and the latter one from 55 — 95 Hz, all including a notch filter at 50 Hz. Therefore, 11,964 EEG fragments constitute the current dataset, counting approximately 2,000 segments per subject, since the others were excluded due to mediocre recording quality. Table 2 summarizes the number of signal segments per subject, contained in the referred dataset.

Finally, Table 3 presents the main parameters of the PL dataset released by Spampinato Spampinato et al. [20].

3.3. Mutual Information-Based Channels Selection

The selection of discriminant channels related to the defined mental task aims to gather channels delivering similar information based on entropy values. In the recent literature, similar methods have been proposed to select contributing channels, particularly in [36,40,41]. In this sense, let $M=[1,2,i,\cdots ,j,\cdots ,128]$ be the finite channel set provided by the dataset, $A_i$ and $B_j$ two probability distributions of channels i and $j\in M$. The Kullback-Leibler Divergence (KLD) assessing how far a signal joint distribution of i and j channels is from the probability distribution of their products is calculated as,

$$\mathrm{KLD}(A_i\parallel B_j)=\sum _{a\in M}{A}_i\left(a\right)log\frac{A_i\left(a\right)}{B_j\left(a\right)},$$

(2)

where $A_i\left(a\right)$ is the occurrence probability of the $a^{th}$ information. Thus, the MutIn between channel pairs is found by evaluating the KLD as follows:,

$$\begin{array}{cc}\hfill I(M_i;M_j)& =KLD(P(M_i,M_j)\parallel P\left(M_i\right)P\left(M_j\right)),\hfill \end{array}$$

(3)

where $P\left(M_i\right)$ and $P\left(M_j\right)$ are signal distributions of $M_i$ and $M_j$ channels, respectively, and $P(M_i,M_j)$ their joint distribution. Generally, calculating MutIn using Equation 3 leads to two cases:

• The case where $M_i$ and $M_j$ are independent, therefore,

  $$\begin{array}{cc}\hfill I(M_i;M_j)& =KLD(P(M_i,M_j)\parallel P\left(M_i\right)P\left(M_j\right))=0\hfill \end{array}$$

  (4)
• In other cases, $M_i$ and $M_j$ channels share the totality of their respective information. Thus,

  $$\begin{array}{cc}\hfill I(M_i;M_j)& =KLD(P(M_i,M_j)\parallel P\left(M_i\right)P\left(M_j\right))=H\left(M_i\right),\hfill \end{array}$$

  (5)

where $H\left(M_i\right)$ represents the self-entropy of channel $M_i$. In this work, making a subset of 64 discriminant channels constitutes a method contribution compared to the state of the art, as will be discussed in Section 4.4. Therefore, when computing Equation 3 according to Algorithm 1, subsets of discriminant channels are constituted by finding maxima cross-entropy values for each considered channel combination.   

Algorithm 1:MutIn algorithm-based-discriminant channels selection

    The channels selection step typically involves the whole dataset signals. This empirical approach makes a constituted discriminant channel subset more representative of all subjects’ signals. Also, this strategy helps for the cross-subject results comparison purpose. From Algorithm 1, lines 3 to 6 calculate entropies combining two-by-two the 128 channels. Next, a channel combination having a high entropy value is used on lines 10 to 20 to make n-channel combinations by adding one discriminant channel in each iteration until the last 64-channel combination is obtained.

3.4. Enhancing Signals by Minimum-Norm Estimates Algorithm

In the literature related to EEG signal processing, the MNE algorithm and its variants have been used more for brain source localization [42,43], estimation of functional connectivity between different brain cortex [21], and EEG inverse problems [44] than for signals preprocessing [45,46]. Typically, the MNE-Python preprocessing pipeline allows EEG quality assessment provided by selected channels. As a result, EEG segments are extracted using band-stop, band-pass, low-pass, high-pass, and notch filtering [21].

In the present work, semi-automatic functions of MNE-Python were implemented to exclude contaminated EEG data and reduce artifact attenuation. Algorithm 2 summarizes the relevant steps in preprocessing EEG data by MNE.   

Algorithm 2:The MNE steps implemented to enhance EEG data.

1 Get EEG data from selected channels 2 Handle poor channels providing extremely noisy data to be usable, based on good signals delivered by other channels. 3 Discard erroneous data gaps and spans. 4 Calculate the variance of the data. 5 Remove the mean and scale to the unit variance to standardize features. 6 Create epoch of data. 7 Average epoch to obtain evoked responses.

The class mne.decoding.Scaler , which includes the steps from 1 to 5 of Algorithm 2 was specifically utilized. This class estimates the mean ($\mu$) and variance ($\sigma$) for each channel by utilizing data from all epochs and time points. That is, the $\mu$ and $\sigma$ of a given $n^{th}$ training sample are estimated as following:

$${\mu }_n\approx \frac{1}{N}\left(\sum _{i=1}^N{x}_i\right),\mathrm{and}$$

(6)

$${\sigma }_n^2\approx \frac{1}{N-1}\sum _{i=1}^N{(x_i-{\mu }_n)}^2,$$

(7)

where N is the number of epochs, and i denotes the time points of each epoch. Therefore, each feature is independently centered and scaled by computing the corresponding statistics on the training set’s samples. By setting $\mathit{scalings}{=}^{\prime }{\mathit{mean}}^{\prime }$, all other parameters of the previously mentioned function were used with their default values, as explained in the implementation documentation [21]. Next, data are fit by standardizing them across channels and then transformed, coupled to their labels to return a new version of epochs data as the output. Hence, $mu$ and $\sigma$ are utilized on data by using the MNE’s fit_transform() function. Finally, the obtained epoch is averaged across channels to obtain evoked responses, covering steps 6 to 7 in Algorithm 2.

3.5. The Implemented Classifiers

Two deep learning architecture models are considered, because of their respective performances in processing EEG signals and the outstanding number of referenced works, namely, the EEGNet and a hybrid CNN-LSTM networks [23,47,48].

3.5.1. EEGNet Network

Built in Keras and Tensorflow [49,50], the EEGNet is a compact convolutional neural network proposed by Waytowich et al. [23] for EEG signal processing and classification. The EEGNet architecture takes its use advantage by combining three convolutional layers. That is, a temporal, a depthwise and a separable convolution layer, as illustrated in Figure 3.

Epochs from the MNE block are convolved by the Conv2D (block 1) where frequency filters are applied. After that, each feature map in the Depthwise Conv2D layer (Block 2) is processed by spatial filters to determine its intrinsic properties. Depthwise and pointwise convolutions are executed independently in the Separable Conv2D layer (Block 3) before being combined. Finally, the classification layer (Block 4) evaluates the probability for an output EEG segment to belong to one of the 40 input labels. To do this, the Exponential Linear Unit (ELU) function is used to activate the Depthwise and Separable Conv2D layers, as follows:

$$f\left({\overrightarrow{x}}_i\right)=\left\{\begin{array}{cc}{x}_i& forx\ge 0,\hfill \\ \alpha (e^{x_i}-1)& \mathrm{otherwise},\hfill \end{array}\right.$$

(8)

and for its part, the dense layer is activated by the Softmax function, given by

$$\sigma {\left(\overrightarrow{x}\right)}_i=\frac{e^{x_i}}{{\sum }_{j=1}^{40}{e}^{x_j}},\forall \overrightarrow{x}={[x_1,x_2,\dots ,x_{40}]}^{⊺},$$

(9)

where $x_i$ and $x_j$ represent the input and output sequences of time points, respectively. The hyper-parameter $\alpha$ controlling the saturation point for negative inputs is set to 1. Specifically, the Conv2D layer featured eight temporal filters (F1) with 320 network parameters, the Depthwise Conv2D layer included ten spatial filters (D) with 4320 parameters, and the Separable Conv2D layer possessed 7680 pointwise filters (F2) utilizing 456 parameters. These filters’ values were set up considering the data structure (sampling frequency, length of samples per task and subject, etc.) and the outcomes of preliminary training tests seeking to optimize the classifier. Table 4 summarizes the layers’ main parameters for the proposed EEGNet model.

In addition to this filter setting, the model was built to receive as input 440-time points delivered by the 54 selected channels. Section 4.4 explains the 54-channel selection instead of the 64 targeted. The kernel length was set to 40 to match the number of output classes, with the dropout set to 0.2. Next, the model was compiled using the categorical cross-entropy loss function, the Nadam optimizer, and the accuracy metric, defined in the results section.

3.5.2. The Proposed CNN-LSTM Model

The CNN-LSTM model has proved its efficiency in processing EEG signals for applications-based BCI systems [51,52]. This architecture finds its greatest use in extracting spatial features at the CNN block level. At the same time, temporal dependencies are identified at the LSTM block level, using stronger learning and memory capabilities. Concretely, EEG time points from preprocessing are memorized and forgotten, allowing one to learn more comprehensive features. Because of memory units in the LSTM’s block, the CNN-LSTM model can remember the prior state of the data, ensuring identification based on the current state change pattern. At the end of this hybrid model, a fully connected layer guarantees the labeled output considering an input data sequence. Consequently, Figure 4 presents the proposed CNN-LSTM architecture comprising three CNN layers, two LSTM units, Fully connected, and Softmax layers.

Therefore, the Conv1D_layer1 is configured with 128 convolutions of 3$\times 3$ kernel size while Conv1D_layer2 and Conv1D_layer3 layers contain 64 filters of 3$\times 3$ size. The He initialization algorithm [53] was used to initialize weights based on a uniform distribution, and the dropout parameter was set to 0.2 for the mentioned layers. All convolutional layers were activated by the leaky Rectified Linear Unit (ReLU) function, given by

$$f(\alpha ,{\overrightarrow{x}}_i)=\left\{\begin{array}{cc}\alpha x_i& ifx<0,\hfill \\ x_i& \mathrm{otherwise},\forall \overrightarrow{x}={[x_1,x_2,\dots ,x_{40}]}^{⊺}\hfill \end{array}\right.$$

(10)

where $\alpha$ represent a small positive constant set to 0.005 to compensate negative net inputs with small non-zero gradient. For their part, LSTM layers were configured with 64 and 32 memory units, respectively, to process time point sequences from convolutional layers. At last, the Fully Connected layer contained 54 neurons, and the Softmax layer used 40 neurons to predict the class probability of the output sequence. As in the case of the EEGNet model, the categorical cross-entropy loss function, Nadam optimizer, and accuracy metric were implemented to compile the model. A parameters summary of the proposed CNN-LSTM model is illustrated in Table 5.

3.6. Experimental Settings

The code implementation of both architectures was developed in Python 3.6 using Keras and TensorFlow. An NVIDIA GTX 2080 Ti GPU-equipped Ubuntu 22.04 desktop computer was used to run the entire project. Therefore, to accelerate the learning convergence of the models, the Cyclical Learning Rate (CLR) algorithm [54] was implemented, in addition to helping to avoid local minima in the learning process. The lower and upper bounds of the learning rates were adjusted to ${10}^{-3}$ and ${10}^{-7}$, respectively, by adopting a triangular window, and the step size was set to 8 times the epoch’s total iterations. The EEGNet and CNN-LSTM models were trained at 1000 epochs for comparison purposes, with a batch size set to 440. Finally, the k-fold cross-validation approach presented in Section 4 was used to support the results achieved in this study.

4. Results

The results presented in this section essentially used data presented in Section 3.2, by implementing the k-folds cross-validation method, where k was set to 10. That is, the whole dataset was divided into ten partitions. Consequently, nine partitions were employed iteratively for training, while one partition was used to validate the models’ performance. Specifically, for each of the ten iterations, 10767 samples were used for training and 1197 samples for validation. This validation technique enables the models’ efficiency to be assessed and evaluated for specific data streams or several unpredictable inputs. Additionally, the developed classification approach used the accuracy metric defined by

$$Accuracy=\left(\frac{TP+TN}{TP+TN+FP+FN}\right)\times 100,$$

(11)

where $TP,TN,FP,$ and $FN$ represent the true positive, true negative, false positive, and false negative, respectively. $TP$ corresponds to each x feature correctly assigned to label X while $TN$ represents each x feature of other labels than X unclassified to label X. For its part, opposite to FN, FP is related to all features misclassified to label X. Lastly, the effectiveness of the suggested approach was also assessed using the confusion matrix metric.

4.1. Results Related to Channels Selection

The first step of the developed method consisted of selecting discriminant channels from the 128 provided by the dataset. This study suggests minimizing the number of channels while preserving good classification accuracy since all current deep learning-based studies use all 128 available channels with only a focus on improving classification accuracy. For embedded BCI systems, that have severely constrained computing resources and low power consumption, this accuracy-data size trade-off is essential [55,56]. Therefore, this work proposed using less than half of the available channels by developing Algorithm 1 to select more discriminating channels. Table 6 presents the results achieved in selecting channels.

The developed approach allowed the selection of eight channels in the brain’s parietal cortex and seven in the occipital and parietal-occipital areas. However, only two channels were found to be discriminating in the frontal-central cortex, compared to three in the frontal-central-central area. In sum, EEG signals from the 54 selected channels were considered for the preprocessing step by the MNE method.

4.2. Results of Preprocessing by MNE

Each EEG segment of the 54 selected channels was enhanced using the MNE algorithm. Therefore, by computing Equations 6 and 7, the length of epochs data was maintained to 440 at the output. As an illustration, Figure 5 presents EEG segments from Subject 4 for a selected group of discriminant channels, before and after implementing the MNE algorithm.

Thus, the new version of epochs data illustrated in Figure 5(b) conserved the class labels of initial data but artifacts were removed through the MNE’s mean scaling procedure. In sum, the data matrix of 11,964 time-point segments from 54 selected channels, containing 440 samples at maximum, was used as input to the classifiers.

4.3. Results Related to EEG Segments Classification

EEG time-point segments were classified according to two approaches. The first one uses a total length of 440 samples. In this sense, Table 7 presents the results achieved by classifying data with the aforementioned models. The classifiers’ accuracy is reported for every $k^{th}$-fold iteration, alongside the number of segments utilized for testing and training. The average accuracy of 93.2 and 88.2% were reached in classifying data into 40 labels, by using the EEGNet and CNN-LSTM networks, respectively.

The second approach considers different sample lengths to evaluate the signal stretches with data more enhanced by the preprocessing step.

Therefore, Table 8 considers nine data time intervals between 20 and 440 samples. This stretch cutting was motivated by the following considerations. On the one hand, looking to evaluate the length of 440 samples in small stretches, leaving out data samples at one extremity. On the other hand, to compare the outcomes of the suggested method with those of the state-of-the-art (SOTA), the selected stretch lengths were assumed. Thus, the classification accuracies reported in Table 8 are averaged from those achieved by each k-fold iteration. The best average accuracy of 94.8% was reached by processing data in 360 - 440 ms, while the lowest of 87.2% was obtained for data in 20 - 240 ms with the EEGNet model. For its part, the CNN-LSTM architecture best performed the classification task by processing the 360-440 ms stretch (89.8%) rather than for the 20-240 ms one (81.3%). In summary, the developed strategy of samples’ stretch-splitting enabled different outcomes, a pertinent observation discussed in the next section. Furthermore, Table 9 and Table 10 present the summary results of the confusion matrices classifying data segments according to each class label. The diagonal results of the confusion matrix, displayed in the tables afterward, illustrate the exactness of predicted versus true labels for a specific output class.

Therefore, the CNN-LSTM model outperformed the data class labeling in the 240-440 ms stretch, followed by the 360-440 ms, 130-350 ms, and 130-440 ms stretches, by achieving accuracies of 93, 92, 91, and 90%, respectively. On another side, accuracies of 88, 87, and 86% were obtained by processing stretches of 360-440 ms, 130-440, 130-350 ms, and 240-440 ms, in this order, with the CNN-LSTM classifier.

4.4. Discussion

One of this work’s important contributions is optimizing the number of contributing channels while ensuring accurate data classification for BCI applications. A subsequent check of discriminant frames was introduced to deduce signal lengths more enhanced by the MNE preprocessing. Regarding the number of channels, this work aimed to use less than 50% of the available channels. This is to deal, on the one hand, with one of the current big data processing issues for embedded BCI applications, where computational resources are constrained, for instance [57]. On the other hand, selecting discriminating channels allowed to minimize the influence of channels whose intrinsic characteristics do not contribute significantly to the classification process, based on the information conveyed [58]. Thus, Algorithm 1 was configured to select 64 channels from the 128 available, based essentially on the mutual information evaluated using the cross-entropies. After configuring the classifiers, the first tests aimed to set Algorithm 1 for a limit of 60 channels, assess the outcomes, and then compare them with those achieved with 64 channels. The differences between outcomes were minimal compared to the large amount of data to be conserved in the processing chain by considering the 64 channels. Comparing the outcomes of 55 channels with those of 60 and 64 channels resulted in the same finding. However, by proceeding with the same approach between 50 and 55 channels, it was uncovered that 53 of the ones selected provided a larger difference than 54 channels, as illustrated in Figure 6.

Consequently, for the remaining processing steps, the concluding selection of 54 channels was determined. Therefore, from the 54 channels selected, 33 are located in the parietal-occipital cortex and 16 in the central cortex. In other words, the mental task of image visualization produced a greater neuronal effect on the motor and visual cortices than in others, as reported by Zheng and Chen [30].

Next, preprocessing by MNE allowed for obtaining the data enhanced, and more suitable for the classifiers. Table 11 presents the obtained results with and without MNE’s preprocessing block integration.

In the case of the EEGNet network, an average relative benefit of 12.8% was evaluated by implementing the MNE’s preprocessing stage, whereas the CNN-LSTM classifier for its part, enabled achieving a relative gain of 13.9%.

The final step of data classification was carried out by the EEGNet and CNN-LSTM models. The better performance of the EEGNet architecture compared to the CNN-LSTM model was observed, based on the metrics presented above. Essentially built around distinct architectures, the CNN-LSTM model required roughly 107278 parameters whilst only 54632 parameters were used to configure the EEGNet network, almost half the number of parameters utilized by the CNN-LSTM model. This finding gives credence, in addition to the performance achieved, to the EEGNet network for embedded BCI applications, for the above-mentioned reasons.

When comparing the outcomes obtained in this work with those reported in related literature, essentially based on handling the PL dataset, the overview observation is that all use data provided by the 128 channels, as summarized in Table 12. In contrast, in this work, the number of channels was reduced by about 57.8% compared to that used by the references. This is one of the key contributions of the proposed method, placing this last one explicitly in the field of embedded BCI systems.

In the last column of Table 12, classification accuracies are compared. Spampinato [20] and Nandini [32] obtained accuracies of 82.9 and 81.59% by implementing RNN/CNN and STFT+EEGCapsNet models, respectively. The proposed approach with both architectures better performed the class labeling task than the evoked works. On the other side, Xiao [30,31] and Nastaran Khaleghi et al. [33] reported accuracies of 99.50, 97.13, and 98.4% by using Bi-LSTM-AttGW, LSTMs_B, and FC-GDN architectures. Their achievement exceeds the outcomes obtained in this framework by an average of 3.5%. However, the accuracy discrepancy compared to the benefit recorded utilizing fewer than half of the available channels is thereby acceptable, considering further that the recommended accuracy for BCI applications is valued above 70%. In other words, processing data provided by the whole set of channels to improve the average accuracy only of 3.5% seems to expend computational resources excessively than optimizing the number of useful channels. Therefore, this work proposes a reliable alternative for embedded BCI applications, offering a suitable trade-off accuracy-computational time.

5. Conclusion

This section is presented in two steps. In the first one, the conclusion of this work is provided by summarizing the interpretation of the results obtained and the developed method scope. In the second part, upcoming work is contemplated based on the results achieved, considering this achievement as the initial phase of an ambitious project to be developed.

5.1. Conclusion

This work aimed to classify EEG visualization signals into 40 labels, that is, dealing with the recent EEG visual multiclass classification challenge, by using a smaller number of channels while maintaining reliable classification accuracy. Public data from the PL dataset including 11964 EEG signal segments were used for the experiment. To be more specific, a set of 54 discriminating channels was built using a channel selection approach based on mutual information. Next, data were enhanced using the MNE method. In the last stage, the EEGNet and CNN-LSTM architectures served as classifiers to label data according to the defined classes. The results achieved demonstrate a good performance of the EEGNet classifier over the CNN-LSTM, by reaching the highest accuracy of 94.8%. Compared with the models proposed in the related literature, this work incorporates the trade-off between the classification accuracy and the number of channels. The latest is a more desired criterion in the implementation of embedded BCI systems based on EEG signals. However, the results presented in this work are constrained to the configuration set in the preprocessing and classification blocks. Also on the use of data provided by the PL database, exclusively. The source codes of this project are available to the public at https://github.com/Tatyvelu/EEG-Visual-Multiclass-Classification.

5.2. Forthcoming Work

The challenge of database availability with more than 40 classes, similar to the PL dataset, motivates forward a multiclass database construction, consisting of EEG signals from fruit sequences visualization. Such a dataset would be useful for implementing the EEGNet model in the NAO robot to assist people in fruit type recognition.