Analysis of Relation Between Brainwave Activity and Reaction Time of Short-Haul Pilots Based on EEG Data

1. Introduction

Safety is an important aspect of the modern airline industry. Among many factors that influence the proper execution of the flight process, pilot performance is one of the most crucial.

Conducted surveys point out that pilot fatigue was recognized as a probable cause of 21-$23\%$ major aviation accidents [1,2]. The importance of fatigue counteraction is confirmed by aviation organizations like National Transportation Safety Board which over the past 25 years created over 200 recommendations related to pilot tiredness [3]. Noteworthy, based on studies conducted in 1980 and 2017 we can conclude that despite the effort put into a limitation of malicious effects caused by pilot tiredness, the tendency seems not to change significantly over the years[4]. What is more, a survey conducted on the group of short-haul pilots points out that over 75 % of them claimed that they were subject to significant fatigue [5] and that over 70 % of corporate pilots claimed that they fallen into micro-sleep during various phases of the flight [6]. Such a state is related to the reduction of the response abilities to external stimuli [7] as well as the degradation of cognitive tasks [8].

In order to assess the performance of the pilots during short-haul flights, a set of experiments was conducted during which participants were flying in simulated conditions of a professional FNTP II class simulator. During simulated sessions subjects’ brain activity was measured with an EEG device.

As a result of the brain cells activation the bioelectric field sources are generated [9,10,11]. These electric fields are transmitted and propagated through the biological tissues of the brain. Such phenomena is known as the volume conduction [10,12,13]. The electric currents produced as a result of neuronal activity are therefore measurable at the surface of the scalp with the dedicated sensors. A very popular method of brain signal recording is the electroencephalography (EEG) [9,10,11]. In this method the measurement sensors, referred to as electrodes, are placed in specific locations over the scalp. These electrodes record the changes in electric potential caused by neural activity. Because EEG signals are characterized by low amplitudes (normally in the range from $0.5$ to $100\mu V$) and bandwidth mostly located below $100Hz$, they are highly susceptible to noise and artifact disturbances. Therefore, differential measurement configurations (uni- or bipolar) are common as they help to decrease the contamination of data. As a result of EEG measurement the electroencephalogram (EEG) is obtained. In the EEG signal some characteristic frequency bands can be distinguished. These are referred to as: Delta (below $3Hz$), Theta ($3-7Hz$), Alpha ($8-12Hz$), Beta ($13-29Hz$) and Gamma (over $30Hz$) [10,14,15]. It is worth mentioning that the frequency limits of specific waves are conventional as there is no proper way of determining their exact values.

During a sleep-related state, the power of the signal in lower frequency bands increases [11,16]. Studies have shown that deep sleep is mostly related to the Delta range. These waves are mainly active during stages III and IV of the non-rapid eye movement sleep (NREM) and are often used during sleep depth assessment [11,17]. There has been reported a relationship between the spectral power of theta activity and memory workload during cognitive tasks (i.e. visual processing) [18,19]. It was additionally observed, that theta power depends on the complexity of the stimulus [20]. Additionally, according to some research [18] processing of previously unknown information leads to an increase of theta activity over fronto-central regions of the brain cortex [21]. On the other hand, both known and simple information could lead to decreases in its power [18]. The alpha activity occurs during states of wakeful relaxation and drowsiness. Additionally, these rhythms can be induced by closing eyes [11,15]. Alpha waves are also often related to the activity of the visual cortex [15,22]. Despite being mostly associated with states of relaxation, according to some research alpha rhythms may increase during some attention tasks [15,19]. In addition, experimental results revealed that as a person fatigues, slow theta and alpha wave activity increases over the entire cortex [23]. Beta states are frequencies associated with normal waking consciousness [11]. These rhythms are present during tasks that require high alertness. An example of such a task could be maintaining the attention on a target during attentive looking at a visual stimulus [15,24]. Interestingly, the studies examining the nature of the relation between the power of these rhythms and cognitive workload show diverging results [19]. An interesting interpretation of an increased frontal region beta activity in fatigued states has been proposed by Craig and others [23]. Increased beta activity has been attributed to the brain’s attempts to maintain vigilance levels that are supposed to compensate for decreased brain capacity in fatigued subjects [23]. The role of Gamma waves is still an active topic of research. Interestingly, signal energy of the Lower Gamma band (32–36 Hz) in the Frontal Lobe (AF3 and F8 electrodes) was the most commonly selected feature for the purpose of prediction of pilot’s reaction time based on EEG signals in a recent study [25].

In order to ensure the reproducibility of the conducted EEG research the electrodes are usually located and labelled in accordance with some universally accepted standard [11]. Such a standard must be designed in a manner that allows the best possible coverage of the functional areas of the brain (i.e. Brodmann’s Areas). As an example, electrode montage systems such as the standard 10-20, as well as its extensions such as 10-10 and 10-5 can be mentioned [26,27]. Presented in Figure 1 are the locations of the electrodes in the 10-10 configuration [27].

The goals of this research were related to the assessment of the relation between the bandpower of pilot’s EEG signal and reaction time with specific brain activity during short-haul flights. The achieved results will lead to a better understanding of cognitive activity underlying the decision–making processes that occur during events that are time sensitive and require immediate reaction. Proper identification of factors leading to a decreased performance in such actions is critical. Potentially it could drastically increase the safety of flights. Additionally, early detection of changes in mental activity that are associated with drowsiness, micro–sleeps and lack of focus could allow applying adequate precautionary measures in advance. Finally, obtaining the ability to predict the pilot’s intended action before the execution of voluntary movements required to perform it, could be utilized in various neuroadaptive technology solutions [28]. One of many extremely appealing examples of such systems are the cognitive cockpit systems [29,30,31].

In this research, an experiment was proposed that allowed for the measurement of the reaction time of subjects exposed to unexpected events during 1-hour long sessions resembling real-life plane piloting scenarios. During the experiments, the electrical activity of the subjects was recorded. For that purpose low-cost and portable Emotiv EPOC+ Scientific Contextual EEG devices were used. Especially, the relationship between the duration of the experiment and response time as well as changes in frequency–specific brain power was examined. Additionally, the correlation between response time and power generated independently in theta and beta bands was assessed. Finally, the possibility to detect relative changes in brain power related to the appearance of visual stimuli or performing an action was evaluated. This work complements previously conducted preliminary research during which the effect of the extended duration of performing monotonous maneuvers on drowsiness and mental fatigue was examined [32]. In addition, previous research has demonstrated that it is possible to predict reaction times on the basis of EEG data [25]. Results of another study confirmed the possibility of using EEG-based BCI systems in cognitive cockpit solutions [33]. This has been achieved by presenting an accurate machine learning model allowing for the discrimination between states of brain activity related to idle but focused anticipation of visual cue and reaction to this cue. The purpose of this work is to provide a detailed assessment and analysis of the nature of the mechanisms that enabled the successful utilization of EEG data for the prediction and classification of participants’ activity patterns.

2. Materials

2.1. Description of experiment

The purpose of the experimentation phase was to measure the activity of the human brain, during the simulated session of short-haul flights, with the autopilot activated, during which visual cues were displayed randomly on the main screen of the simulator.

Participants were selected from the group of people aged between 20 – 35. All participants claimed that they were well-rested before the session, and all of them gave consent to the utilization of outcomes obtained during the experiment for the purpose of scientific research. During the experimentation phase 8 people were examined. Data were gathered at the same time of the day (around 12 a.m.) and it was ensured that no external factors had an influence on experiment participants. Each session took around 1 hour and was preceded by a short introduction and installation of EEG device on the pilot’s head. Participants were seated in FNPT II class simulator. The task of the participants was to act as regular pilots therefore, they have to observe cockpit instruments as well as scan the surrounding of the plane. They were instructed to maintain awareness as well as stay focused in order to be able to instantly react to visual cue event occurrence. When a visual cue was displayed on the screen participants were instructed to press a specific button immediately. In order to minimize the time for visual cue reaction, the button was placed in a location that does not require any additional movements of the pilot’s body besides their fingers.

In order to obtain coherence between consecutive experimental sessions, a simulated flight on the route between Frankfurt and London was recorded and the same section of the flight was presented to every participant of the experiment. Both the terrain over which the flight took place as well as cockpit instruments were registered. During this flight autopilot was activated. The flight took place at an average altitude of 6000 feet. In order to simulate flight with the autopilot activated, take-off and landing were removed from registered material. In addition, the whole flight, which was presented to the participants, took place over land. It is worth mentioning that the sounds of engines were also generated in the cockpit.

Visual cues were displayed randomly with normal distribution characterized by $\mu =2.5$ minutes and $\sigma =1$ minute. The variance was introduced in order to prevent the habituation of the human brain to regular patterns. In addition, for each pilot distribution of visual cues in time was the same. The visual cue was represented by a solid, red coloured box that overlaps 75 % of the main simulator screen (that is responsible for displaying the terrain ). The consent of the bioethics committee was obtained, which allowed conducting of such experiments.

2.2. Flight simulators

During experimentation sessions a professional Flight Navigational Procedure Training II (FNPT II) class simulator was utilized (Figure 2). The simulator was built by SoftekSim company, it is based on Lockheed Martin Prepar3D software that reflects the Cessna 172RG plane model and it passed QTG tests. The flight simulator consists of a fully enclosed full-size cockpit, that faithfully reproduces the internals of the Cessna 172RG equipped with a glass cockpit. It is characterized by a 180-degree panoramic view of the environment, that is generated by three projectors. The simulator is located in an especially designated room (Virtual Flight Laboratory located at the Silesian University of Technology), without any windows and with black walls thus no external stimulus can reach the pilot.

2.3. Emotiv Epoc+ Headset

The EEG data used in this research was recorded using the Emotiv EPOC+ Headset. This device uses a sequential sampling method with a rate of 128 SPS and provides signals of 14 bit ($1LSB=0.51\mu V$) resolution. The device has a built-in digital 5-th order Sinc filter and notch filters at 50 Hz and 60 Hz, implemented in order to enhance the quality of the recorded signal. The recorded signals are characterized by useful bandwidth in $0.16-43$ Hz range [34]. Emotiv EPOC+ Headset gives access to 14 EEG channels compatible with the international $10-10$ electrode montage system. The available electrodes are: AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4, with references in the P3/P4 locations [34]. The placement of EPOC+ electrodes with regards to the $10-10$ configuration was marked in Figure 1 [27].

The cerebral cortex of the human brain can be divided into four major lobes: Frontal, Parietal, Occipital and Temporal. Besides the anatomical compatibility, the introduced classification corresponds with the functional classification of the brain’s areas [35]. Presented in Table 1 are the cerebral lobes that correspond to the specific electrodes available in the Emotiv Epoc+ Headset [26,27].

The most important role of the Frontal Lobe is processing tasks associated with planning, motivation, short-term memory and attention. This also involves the discrimination between events and the assessment of consequences associated with performed actions. Additionally, the Frontal Lobe plays a major role in voluntary movement planning and control [26,35]. The Occipital Lobe is responsible for visual processing actions such as focus and identification of stimuli, motion perception, visuospatial orientation and colour differentiation. It also takes part in coordinating motor actions in response to the outside stimuli [26,35]. Processing of somatic sensation as well as integrating sensory information from various parts of the body is generally assigned to the Parietal Lobe [26,35]. The Temporal Lobe is mostly involved in auditory processing both on a low and a high level (i.e. language recognition). Additionally, there are areas of this lobe, that are associated with interpreting the visual stimuli and establishing the object recognition [26,35].

Because of the phenomena known as the volume conduction, fields originating from distant sources of bioelectrical activity are diffused and propagated through brain tissues. As a result, they reach multiple electrodes and mix with the signals produced by local sources. Due to that EEG signals are characterized by very low spatial resolution. It has been shown that the sources within a 3 cm radius of each scalp electrode contribute only partly to the measured signal [36]. The effects of source overlapping are sometimes corrected with the use of spatial filtering methods [37,38]. However, this research follows a methodology applied in our previous work where spatial filtering has been omitted [25]. The benefit of this approach is the consistency of the results obtained over a series of analyses performed on the same data. Additionally, during the experiment it was observed, that EPOC+ electrodes were not always positioned precisely at the $10-10$ locations assigned to them. The reason for that could be found in the fact that the construction of EPOC+ does not allow for any significant adjustments of electrode placement and, depending on the shape and size of the subject’s skull, some inaccuracies can surface. Additionally, some further misplacements could result from adjusting the electrode position for the best quality of the recorded signal. However, it must be noted that said misplacements were mostly of a subtle nature and had never led to the displacement of the electrode outside of the cerebral lobe it was originally assigned to. Considering the aforementioned observations, it was decided to focus on the more general activity related-lobes rather than on the specific locations of electrodes.

The Emotiv EPOC+ Headset system is an inexpensive and practical solution for basic scientific research. However, during the measurements it was observed that recorded data are susceptible to contamination by the artifacts related to muscle movements that occur e.g. during motor actions of limbs, head repositioning or blinking. In order to reduce the impact of such artifacts on the results of performed experiments special precautions were undertaken. All subjects were seated in a comfortable position and instructed to limit their movements as much as possible. Additionally, the reference and event segments were manually evaluated for the presence of artifacts. Trials that were assessed to contain too much contamination were removed from the analysis.

3. Methods

3.1. Digital processing of EEG signals

In this research, EEG signals were analyzed independently, in frequency ranges corresponding to specific brain waves i.e. Theta, Alpha and Beta. Therefore, a method for spectral filtering of these signals had to be chosen. For the purpose of bandpass filtering of EEG data, a zero-phase (non-delaying) filter was applied during offline processing. This is usually implemented by applying a recursive filter to the original signal both forwards and backwards in time.

Let $x\in {\mathbb{R}}^M$ be a recorded, discrete signal consisting of length M and h be the impulse response of the recursive filter. The output $v\in {\mathbb{R}}^M$ of filtering operation performed on x is calculated as in (1).

$$v=h*x.$$

(1)

If $x\left(i\right)$ ($i=1,\dots ,M$) denotes a discrete sample o x, then the operation of flipping the signal can be defined as in (2) [39].

$$\forall i\in \mathbb{Z},i<M:flip\left(x\left(i\right)\right)=x(M-i).$$

(2)

The flip operator reverses the order of samples of a discrete signal x[39]. Considering the above definitions the output of forward-backward filter $y\in {\mathbb{R}}^M$ can be calculated as presented in (3) [39].

$$y=flip\left(h*flip\left(h*x\right)\right).$$

(3)

In this research, a Kaiser Window Finite Impulse Response (FIR) band-pass filter constructed of 466 coefficients was used. Due to their linear phase characteristics, FIR filters are well-suited for biomedical signal processing applications. However, their disadvantage is manifested by non-negligible delays that are introduced to the data as a result of their use. The approach that was implemented in this research allows for benefiting from the advantages of FIR filters, while at the same time, it eliminates problems related to phase delays in offline processing. Besides spectral division of EEG signals, bandpass filtering additionally improves the key characteristics such as Signal-to-Noise Ratio (SNR) and removes distortions caused by the electrical line drift and high-frequency noise.

3.2. Description of data features

The band power features used for the analysis of brain activity during the experiment were extracted from spectrally filtered signals, individually for each measurement channel. Because the mean value of bandpass filtered EEG signal is close to 0 its power is equivalent to its variance. To normalize the distribution of calculated features a logarithm operation is commonly applied [40]. A logarithm of the variance of signal’s amplitude calculated during some time interval is a very popular feature used for the description of EEG signal’s power [40].

The choice of time interval in which the band power features are calculated is an important issue. In this research two segments of the signal for each event were taken into consideration. The reference segment begins $2.5$ s before the visual cue indicating the start of each new event and finishes $0.5$ s before the marker appears. The activity segment consisted of all samples within $4.5$ s time window from the beginning of an event. The purpose of a reference was to serve as a basis for the determination of the level of change that occurred in the mental activity with the appearance of visual stimuli. Additionally, this time segment is representative of idle brain activity. The length of activity segment was selected on the basis of all reaction times achieved by subjects during the experiment as $98\%$ of all results did not exceed $4.5$ s. Visual representation of the segments used in this research with regards to the appearance of the visual cue has been shown in Figure 3.

The reaction times were calculated as a difference between the time of the subjects’ reaction to the visual cue and the time when the cue appeared. Therefore, higher values of this statistic should be interpreted as a slower response to the stimuli. It was assumed that the time required for the signal to be transferred from the keyboard to the logging device is negligible. In neurosciences, it is assumed that the appearance of a stimulus can induce changes in the neuronal activity time-locked to an event, known as event-related potentials (ERP) [13]. The event-related phenomena can additionally represent frequency-specific changes in the EEG which are manifested as either increase or decrease in signal’s band power. This phenomenon is called Event-Related Desynchronization or Synchronization (ERD/ERS) [13]. A metric based on the ERD/ERS was selected to assess the changes in mental activity after the appearance of the stimulus. In their work, Pfurtscheller and da Silva have precisely formulated the assumptions behind the ERD: ’the term ERD is only meaningful if the baseline measured some seconds before the event represents a rhythmics seen as a clear peak in the power spectrum’ [13].

Although calculated in the same manner and with respect to the assumption of ERD/ERS, a metric used in this research will be referred to as Event-Related Change and calculated on the basis of band power of the activity and reference segments as presented in (4).

$$ERC={\displaystyle \frac{log\left(var\left(\mathit{activity}\right)\right)-log\left(var\left(\mathit{reference}\right)\right)}{log\left(var\left(\mathit{reference}\right)\right)}}$$

(4)

4. Results of the Analysis

4.1. Relation between response time and experiments duration

A trend analysis of the experimental data was performed, to determine the existence of an association between the reaction time of each subject to the appearance of the visual cue and the duration of the experiment. For that purpose, a linear function was fitted to data with the time stamp of the appearance of visual cue serving as function input and time of reaction as output. The coefficients of the function were estimated with a simple linear regression method.

To assess the significance of estimated linear regression coefficients a F-test was performed. The null hypothesis stated that all non-constant coefficients of the regression equation are zero. The alternate hypothesis was that at least one of the non-constant coefficients in the regression equation was not equal to zero. With the timestamp of the event being the only explanatory variable, the rejection of the null hypothesis indicated the non-zero value of the slope coefficient. The proposed trend analysis approach was based on the analysis of the sign of that coefficient. For statistically significant results the positive values would reveal a tendency of the subject’s response time to increase over the time of an experiment. On the other hand, negative values would suggest the opposite. The p-values of calculated F-statistics for each subject are presented in Table 2.

The p-values calculated for each subject exceeded $0.05$. Therefore, no reasonable confidence level would be satisfied in this test. Concluding from that, there is no reason to reject the null hypothesis. On that basis, it can be determined that no linear trend changes in response times related to the duration of performed experiments were present. This complies with the visual inspection of experimental data.

4.2. Relation between experiment duration and general brain activity

The goal of this analysis was to determine whether any overall increase or decrease in the subject’s brainwave-related band power in a specific cerebral location, over the course of the experimental session, was present. Such relation was looked for in theta, alpha, beta and gamma waves, as the increase in power in these bands has been notoriously associated with general fatigue. For the analysis, electrodes belonging to the Frontal, Temporal and Occipital Lobes were considered.

For the purpose of this analysis, an approach similar to that proposed for the evaluation of the trend in response times was implemented. A linear regression equation was fit with the time of an event as an explanatory variable. As a dependent variable a smoothed logarithm of signals power in individual bands calculated from the reference ROIs preceding events was used. Data were smoothed with a Simple Moving Average Filter (SMA) of length 128 (1 s time window). The selection of reference segment for this purpose is justified by the fact that these time segments represent idle brain activity. During that time segments subjects were not performing any other mental activity other than focusing on the upcoming event. Therefore, the use of the reference segment is perfectly suited for determining whether any changes in awareness or fatigue occur over the time of the experiment. To evaluate the significance of the linear regression slope coefficient a F-test with $p=0.01$ was performed.

The dominant trend direction for each subject was determined based on partial trends of all 8 powers courses calculated on the basis of all electrodes belonging to a given lobe. If over $75\%$ of electrode trends had the same sign of trend direction, this direction was assigned as dominant, meaning either power increase (denoted as +) or decrease (denoted as −). If over $75\%$ of estimated slope coefficients were recognized as statistically insignificant (i.e. null hypothesis that they are equal to 0 could not be rejected) it was assumed that no significant increase or decrease in power occurred. If none of the above could be determined, meaning that no trend behavior was common for at least $75\%$ of Frontal Lobe electrodes, it was assumed that changes were inconsistent (denoted as $NC$ for Non Consistent). In Table 3, Table 4 and Table 5 presented are the summaries of the subject’s dominant trends of power changes in theta, alpha and beta waves with respect to the Frontal, Temporal and Occipital Lobes. Examination of obtained results reveals that for the majority of subjects statistically significant changes in any of the bandwidths were not observed in that region.

Frequency ranges used to represent specific brain waves for the purpose of the analysis are presented below:

• theta waves: 3-7Hz
• alpha waves: 8-12Hz
• beta waves: 13-29Hz
• gamma waves: 30-69Hz

4.3. Correlation between EEG power of brainwaves and response times of subjects

The Pearson correlation coefficients between the amount of brain power related to specific brainwaves and specific response times were calculated individually for every subject, in order to determine the existence and nature of the relation between these factors. Precisely, the correlation between response time to an event and logarithmic theta and beta powers in activity segment that was present directly after the presentation of the cue were examined independently.

Statistic tests for the significance of the correlation coefficient with $\alpha =0.05$ (for two-tailed critical values) were performed on the obtained results. The acceptance of the null hypothesis was unequivocal with the statement that the population correlation coefficient equals 0. Therefore, in such a situation no association between data could be claimed. Rejection of the null hypothesis would reveal that a non-zero correlation could exist. Once there was some significance confirmed, the sign of the correlation coefficient was examined to reveal it’s nature. The positive, and statistically significant, correlation could potentially mean that the longer the time required for the subject’s reaction, the greater the mental workload generated in the specific frequency range. Similarly, the negative correlation could potentially be attributed to the situation where lower energy in activity segment in a specific band resulted in a slower reaction. In Table 6, Table 7, Table 8 and Table 9 are presented the summaries of subjects who showed a statistically significant (confidence level $\alpha =0.05$) correlation between theta, alpha beta and gamma band power in specific cerebral lobes.

A statistically significant, positive correlation between Theta Power in Occipital Lobe and response time was reported for the preponderance of the participants (6 out of 8). Additionally, the majority of subjects (5 out of 8) had statistically significant, positive correlations between band-power and reaction times in Frontal Lobe for the Theta range and in Occipital Lobe for the Beta range. Half of the subjects had positive correlations for the Alpha range in the Frontal and Occipital Lobes.

Values of correlations calculated for each electrode in each band-power are presented in Appendix B.

4.4. Analysis of frequency-specific Event Related Changes

One extremely interesting aspect of this research was to analyze and determine whether any change in signal power can be observed between signals occurring before and after the same trial event. In order to infer this information, Student’s two-tailed t-test with critical value $\alpha =0.05$ was performed to assess whether the mean value of all ERC, measured for the subject during the session, is non-zero. The null hypothesis stated that no significant Event Related Changes during the session occurred and the alternative hypothesis said that the mean of all ERC during the current session was non-zero. Since ERC values can be both positive and negative, there is a possibility that high relative changes will lead to 0 mean (i.e. in the case of alternating ERC signs), even despite the fact that absolute ERC value may indicate a strong change in band power. Nevertheless, such a situation would imply that there is no consistent pattern of brain power change (increase or decrease) that could be generalized into applicable conclusions. Therefore, rejection of such cases by the proposed statistical test will be desired.

In Table 10, Table 11 and Table 12 presented are subjects who achieved non-zero mean in specific band power and cerebral lobes.

Statistically significant ERC could be observed for the preponderance of subjects in the Frontal Lobe for Theta frequencies. The majority of subjects (6 out of 8) had significant changes in EEG signal power for Beta waves in the Frontal Lobe and, interestingly, in all lobes for the Gamma band. It is worth noting, that for most (5 out of 8) of the participants, significant ERC could be observed for Theta and Beta frequencies in the Temporal and Occipital Lobes and for Alpha in the Frontal Lobe.

4.5. Topographic analysis of Event Related Changes

An investigation of the differences in brain activity patterns corresponding to the reaction times has been performed. Reaction times have been first assigned a rank of either $slow$, $medium$ or $fast$. The ranking has been based on the tertile to which each reaction time belonged. Therefore, reaction times from the first tertile were the fastest for a given individual and assigned $fast$ rank. The third tertile was associated with the slowest reaction times and, consequently, assigned to the $slow$ group. The remaining reaction times (second tertile) were ranked as $medium$. The grouping was performed independently for each subject to avoid bias coming from the individual characteristics of the participants. Differences coming from natural predispositions could have led to the incorrect interpretation of the results (i.e. $medium$ reaction times for one subject might be $fast$ for another one). Once reaction times were ranked, the ERCs calculated in $theta$, $alpha$, $beta$ and $gamma$ EEG frequencies were averaged within the rank and visualized.

Presented in Appendix A are topographic maps of brain activity visualized for different wavelengths. The ERCs are grouped by reaction time tertiles accordingly to the procedure described in this section. In addition to maps for individual subjects Figure A1–Figure A8, an averaged activity map for all subjects has been included in Figure A9. The topographic maps have been generated with MNE-Python package [41].

It can be observed that brain activity patterns of all subjects for $slow$ reaction times are characterized by an increase in the ERC in the left hemisphere in the Temporal Lobe. This applies to almost all frequency ranges analyzed in this work. The most significant, observed activity patterns are described below

• ERC increase in the left hemisphere of the Temporal Lobe in Theta frequencies was found for all subjects, except Subject 2.
• ERC increase in the left hemisphere of the Temporal Lobe (often slightly overlapping with the Occipital area) in Alpha frequencies was found for subjects 1, 4, 6, 7 and 8. Similar activity but shifted to Frontal Lobe was observed for Subjects 2 and 5.
• ERC increase in the left hemisphere of the Temporal Lobe in Beta frequencies was found for Subjects 2, 3, 4, 5, 6 and 8.
• ERC increase in the Frontal Lobe in Gamma frequencies was found for Subjects 1, 3, 4, 5 and 6.

These observations are further supported by the analysis of the average activity patterns for $slow$ rank presented in Figure A9. Especially, the Temporal activity in Theta and Alpha waves and Frontal activation in the Gamma range are clearly defined.

Analysis of topographic maps of brain activity regarding $medium$ reaction times allowed to identify patterns that are specific to groups of subjects. Alpha ERC increase in the left area of the Temporal Lobe could be observed for almost all subjects.The only exception for that was found for Subject 7 (Figure A7) which had only a decrease in ERC in the right area of the Frontal Lobe. Additionally, four smaller cohorts of subjects could be observed that shared similar activity patterns for $medium$ reaction times:

• Alpha ERC increase in the left area of the Temporal Lobe could be observed for almost all subjects.
• ERC increase in the left hemisphere of the Occipital Lobe in Beta frequencies was found for Subjects 1, 2, 5, and 8.
• ERC increase in the right hemisphere of the Frontal Lobe in Beta frequencies was found for Subjects 2, 4, 7 and 8. Subject 6 had similar activity in the central part of that lobe.
• ERC increase in the Occipital Lobe in Gamma frequencies was found for Subjects 1, 3, 4, 5, and 8.
• ERC increase in the Frontal Lobe in Gamma frequencies was found for Subjects 2, 3, 4, 6, 7 and 8.

Interestingly, apart from right Frontal activity in the Beta range and no clear pattern in Theta, observed activities are not observable in the averaged activity maps plot presented in Figure A9.

In general, the right hemisphere seems to be the most involved part of the brain in actions related to $fast$ reactions. The activity is mostly visible in the Frontal Lobe as increases in the ERC. The most outstanding observations have been described below:

• ERC increase in the Occipital Lobe in Theta frequencies was found for Subjects 1, 2, 3, 4, 5, 6 and 8.
• ERC increase in the right hemisphere of the Frontal Lobe in Alpha frequencies was found for Subjects 1, 2, 3, and 5.
• ERC increase in the right hemisphere of the Frontal Lobe in Beta frequencies was found for Subjects 1, 3, 4, 5, 7 and 8.
• ERC increase in the right part of the Frontal Lobe in Gamma frequencies was found for Subjects 1, 3, 4, 5, 7, and 8.

Apart from right frontal activity in Beta and central Frontal in Gamma, patterns described above are not clearly visible on the average activity patterns presented in Figure A9.

5. Discussion

Statistical analysis of estimated trend directions of the relation between the response time of a particular subject and the duration of experimental sessions did not reveal any significant, non-constant tendencies. This observation complies well with the visual inspection of the data. In general, it was noted that all deviations in response time of each subject did not exhibit any time dependence. This remark leads to the conclusion that a one-hour-long experimental session was not sufficiently long enough in order to downgrade or in any other way influence the performance of subjects in the posed task.

It was generally observed that as the experimental sessions progressed, brain power of theta, alpha, beta and gamma waves maintained a steady level in the frontal, temporal and occipital areas of the cerebrum. This may lead to the conclusion that the proposed experiment did not induce states of either drowsiness or fatigue among subjects. Analogous observations were made during the analysis of the response times. This should be attributed to the duration of each experimental session. To summarize both of these conclusions, it must be stated that according to this research, one-hour-long sessions of simulated flight did not contribute in any way to the long-term changes of either band power in any part of the cerebrum or decreased response time of subjects. Therefore, such short sessions can be considered as relatively safe in terms of mental workload, fatigue and drowsiness of pilots.

A positive correlation between reaction time and theta, beta as well as alpha power in the Occipital Lobe was reported for the majority of the subjects. This part of the cerebrum is strictly related to the processing of visual information. Such results indicate that Occipital Lobe was active and involved throughout the experiment. Additionally, the positive nature of the correlations means here that the more power the EEG signal had in that area of the brain after the visual cue had been presented, the longer it took for the participant to react appropriately. For further discussion, let us assume that higher demand for neuronal activity indicates that deciding on the proper action is more difficult from the human brain’s perspective. Based on such an assumption, it can be speculated that higher reaction times were, at least partially, a result of Occipital’s Lobe inability to efficiently process the visual cue. This can be further supported by the presence of Non-Zero ERC of signal power in the Occipital Lobe. Such changes are interpreted as statistically significant differences between the signal’s energy before and after the presentation of the visual queue. A significant elevation of the energy might be a result of either it’s low state before the event or increased demand after the queue. The first scenario might be attributed to lower concentration. This is especially true in the case of Beta waves, since these waves are mostly associated with tasks requiring attention and concentration.

Positive correlations between Theta and Alpha power in the Frontal Lobe and the subject’s response time have been found for most participants. Once again, such results indicate that the higher the power of the EEG signal was in the Frontal Lobe after the visual cue appeared, the longer it took for the subjects to react appropriately. This may be attributed to the reported in the literature relation between memory workload during cognitive processing in this band. This finding is additionally supported by the commonly accepted association of the Frontal Lobe to the processing of various tasks (including those requiring attention) and discrimination between events. In the research, this can be interpreted as assessing whether the observed marker corresponds to the assumed visual cue and on that basis deciding whether to press the button or not.

The positivity of correlations may be attributed to the greater amount of power required in order to leave the state of attentive cue awaiting (i.e. subject became temporarily less focused on the task). Another explanation could relate this phenomenon to the case-specific, harder and more demanding, process of determining the right reaction to be performed (i.e. stronger brain activity was required). Both of these factors lead to an elongation of time devoted to processing and association of visual cue, which directly translates to a delayed or slower response to the event. Concluding, more theta-related or beta-related activity is interpreted as related to more demanding brain processing, which justifies slower response time to stimuli. It must be noted, that the aforementioned factors are not mutually exclusive. Additionally, they are not related to the subject’s general level of fatigue, drowsiness or concentration since other analyses of this data have shown that no significant overall decrease or increase in these states was present. Therefore, it is concluded that these factors are of temporal nature and occur locally around the time of a visual cue presentation.

Detection of frequency-specific Event Related Changes from EEG signals was not a main task of the conducted research. Its purpose was to evaluate the potential of using such data as part of a Man-Machine Interaction solution for cognitive cockpit systems. Therefore, the nature of these changes was not taken under examination. Instead, the focus was placed on determining whether such behavior is present and displaying any characteristics related to frequency and location that can be generalized to a greater group of subjects.

Changes in Theta activity related to an occurrence of an event were detectable for all subjects, especially in the Frontal Lobe. Additionally, they were also present in the Temporal and Occipital Lobes for most of the subjects. All three lobes that were under examination in this research displayed some kind of changes in beta activity for the majority of subjects. Amplification of alpha activity in the Frontal Lobe was also prominent in most subjects. Interestingly, Gamma power changes were consistently observed in Lobes for the majority of the participants. Achieved results prove that performing a reaction-based task is detectable on the basis of EEG recordings. Therefore, such signals have the potential to be used in cognitive cockpit applications.

An interesting observation can be drawn when analyzing the differences between the most common brain activity patterns for $slow$, $medium$ and $fast$ reaction time groups. Left Temporal increase of ERC in both Theta and Alpha waves seems to be most prevalent for $slow$ reaction times. With the improvement of reaction times ($medium$ range), it can be observed that activity in Theta is silenced. At the same time, temporal activity in Alpha remains clearly visible. Finally, for the fastest reaction times, Alpha activity shifts towards Beta and Gamma in the right hemisphere in the Frontal Lobe. Additionally, activation in the Occipital Lobe becomes visible in the Theta range. Based on these observations it can be claimed that both, the greater involvement of the right hemisphere in the Frontal Lobe and lesser temporal activity in the Alpha range, combined with greater activation of the Occipital Lobe in the Theta range, contributes to the faster reaction times of the participants. This is perhaps the most significant founding of the topographic analysis of brain activity conducted within this research. Another interesting observation can be drawn for Gamma frequencies. For the slowest reaction times, Gamma activity is mostly prevalent in the central area of the Frontal Lobe. While this remains the case for the reaction times ranked as $medium$ in Section 4.5, Gamma wave energy starts increasing in Occipital Lobe as well. For the fastest reaction times, the Gamma activity remains strongly expressed in the Frontal Lobe. However, it can be observed that the activity patterns become shifted towards the right hemisphere. Based on these observations, it can be further concluded that greater activation of the right hemisphere in the Frontal area of the cerebrum is associated with faster reaction times.

6. Conclusion

Presented research provided inserting insights into the activity of the human brain during prolonged and repetitive tasks requiring steady maintenance of attention and fast reactions.

Perhaps the most important finding reveals that both, the Occipital and Frontal, Lobes were mostly involved and active when reacting to the presented visual queue. This was especially significant in the Theta and Alpha bands of the EEG signal. Based on the well-documented knowledge of the roles of these areas of cerebrum and brainwaves, these findings can be easily traced to and associated with visual queue processing, decision making and focus-maintaining processes. The discovery of positive correlations between frequency-specific brain power and the time required for reacting to a visual cue is a very interesting finding. According to the best of this article’s Author’s knowledge, such phenomena have not been exhaustively explored and interpreted, especially in the context of aircraft safety and operations.

The fact that, apart from some minor exceptions, the mental activity and reaction times of almost all subjects did not show any significant signs of progressing tiredness, drowsiness or mental fatigue was compliant with subjective self-assessment of their own state. This leads to an important observation, that one-hour-long sessions of flight attentive monitoring interrupted by occasional fast response demanding tasks do not affect mental states related to fatigue and tiredness of participants. Therefore, for tasks like this, or in some way similar, such duration can be considered as safe. Conducting research with longer experimental sessions and more diversified stimuli (i.e. auditory) are necessary steps that will allow to investigate this subject further.

Event-Related Change of EEG signal proved to be an effective metric. It can be successfully used to assess the existence of changes in signal power caused by the appearance of visual stimuli. Since such signals are related to mental state, the ERC metric can be additionally considered as a meaningful descriptor of changes in brain activity. A high number of observed significant ERCs in the Theta band for the Frontal Lobe and in the Gamma band for all three examined lobes are the most consistent results of this aspect of the presented research. Focusing on these bandwaves and cerebral areas will be a natural extension and continuation of this research.

Analysis of the common activity patterns presented in Section 4.5 leads to an interesting observation that involvement of certain parts of the cerebrum results in different performance in time-restricted tasks. This is an intriguing subject that, if further analyzed, might help to better understand the nature of the components of fast and slow reflexes and reactions.

This work focuses on the analysis of participants’ brain power, reaction times and their relation. Authors try to provide an interpretation of the results in neurocognitive and physiological contexts based on the known theory, most recent research and the nature of the experiment. However, it must be kept in mind that such conclusions cannot be accurately validated as it is impossible to assess how difficult was the specific repetition of the task for the subject or whether the participant felt unfocused shortly before the event occurred. Trying to assess that during the experiment (i.e. through a survey or questionnaire) would disturb it and greatly affect it’s outcome. At the same time, asking the subject about individual trials at the end of the experiment wouldn’t be reliable.