Sensing Cognitive Responses Through a Non-Invasive Brain-Computer Interface

1. Introduction

Mental state, stress, work capacity, time and ways of rest are individual factors that are of great importance and to which more and more attention is paid. Stress and mental health can lead to physical illness, depression, decreased concentration and productivity at work, and affect relationships with other people. Proper rest and time management contribute to a better quality of life and overall well-being. In recent years, systems for assessing the current psychological state have been gaining wider applications. They could help diagnose and monitor mental disorders, such as depression and anxiety, help organizations assess stress levels and psychological balance of employees, and serve to more fully identify the needs of students, allowing educators to provide appropriate and adequate support. Questions about mental stress are becoming more and more relevant and are of interest to various spheres of public life, such as industry, medicine, education, security, sport and many other sectors are interested in how the brain responds to different types and durations of mental stress. To study mental activity, multimodal systems are often used to measure different parameters when solving a cognitive task [1]. Some of these parameters are important physiological indicators such as EEG, pulse, oxygen concentration in the blood, the change of which can make it possible to assess stress factors and predict their impact on the current mental state. Stress assessment could provide opportunities for preventing undesirable situations in the workplace, for optimizing various work, learning processes, etc. [2]. Brain-computer communication systems have been created for various scientific and medical applications, such as restoring the quality of life in subjects with motor disorders or neuromuscular injuries, as well as for improving various professional activities.

Brain-computer interfaces (BCIs), based on electroencephalography (EEG), are a non-invasive method for capturing brain activity through surface-mounted scalp electrodes. EEG is safe, offers high temporal resolution, ideal for real-time applications. It is widely used in BCIs systems due to its cost-effectiveness, portability, and ability to detect rapid changes in neuronal activity [3]. To improve performance, multimodal BCIs are being developed by combining EEG with other physiological signals. In order to investigate stress and cognitive load in complex and dynamic environments, studies using a combination of techniques such as functional near-infrared spectroscopy (fNIRS), electroencephalography (EEG), electrocardiography (ECG), or galvanic skin response (GSR) have been conducted [4]. Integrating EEG with galvanic skin response (GSR) in brain-computer interfaces (BCI) is a well-established approach for capturing both neural and autonomic physiological signals simultaneously. GSR, also known as electrodermal activity (EDA), measures skin conductance, which is modulated by sympathetic nervous system activity and reflects emotional arousal or stress levels, independent of conscious control. The integration of EEG with galvanic skin response (GSR), which measures changes in skin surface resistance, with infrared thermography, which measures changes in facial skin temperature, makes monitoring multimodal. This allows for the measurement of autonomic nervous system activity, providing additional physiological information that may improve the accuracy of assessments of cognitive and physiological status and the ability to assess various emotional and physiological states.

The non-invasive BCIs systems, in addition to the main data channel directly from the brain, often use parallel multimodal sensing to improve accuracy and higher reliability. Examples of this are the combination with functional near-infrared spectroscopy (fNIRS), GSR or infrared thermography. Their combination is done to simultaneously utilize their individual strengths such as high temporal resolution, recording of local blood perfusion, etc.

Many BCI systems have been created to assess and track performance on various cognitive tasks [5]. A part of them explore auditory and motor imagery, navigation, and more [6,7]. There are those that are used for training [8,9], for controlling various mechanical devices, such as simulated wheelchairs, robotic devices, prostheses, etc. [10,11,12].

The experiment examined the influence of cognitive stress (mental workload) on a person's physiological reactions. This classic paradigm in psychophysiology is related to emotional regulation, attention, and response to stressors. A multimodal, non-invasive BCI composed of five parallel measured parameters was used to conduct the study: EEG, heart rate (HR), galvanic skin response (GSR), facial surface temperature, and oxygen saturation (SpO₂). The goal was to test whether these indicators, observed simultaneously, could distinguish the state of rest from a state of mental strain and whether they can distinguish between tasks of different difficulty.

2. Subjects, Materials and Methods

2.1. Subjects

The study was conducted under the oversight and approval of the Ethical Committee of TU- Sofia. Healthy volunteers were recruited by placing adverts at the premises of the host institutions. Informed consent was obtained from all subjects involved in the study. Demographics - age X-Y, sex etc. Ten healthy volunteers participated in the study (7 male & 3 female, average age 43.8 years, all right-handed) after signing an informed consent.

2.2. Study Protocol

The experiments are composed of five parts.

2.2.1. The first part is an initial entrance rest Rest 1 lasting three minutes.

2.2.2. This is followed by the first cognitive task – the Stroop test, which also lasts three minutes.

2.2.3. Then there is a two-minute break.

2.2.4. The next part is Subtraction – an arithmetic task with a higher cognitive workload.

2.2.5. The last part is again a rest Rest 3 lasting three minutes.

Measurements were taken every thirty seconds, with the first recording being at the beginning of the first break and the last being at the end of the third break. This sequence allows for alternating cognitive load with recovery periods, which facilitates sensing and comparing alternating physiological states. The experimental setup is designed to provide high-quality signal, minimize noise and artifacts, and reproducibility of experiments, all combined with minimal discomfort to participants. These qualities make it very suitable for conducting research in the field of non-invasive brain-computer interface.

2.3. Stroop Test

The Stroop task is a neuropsychological test of attention and cognitive control that assesses the ability to inhibit cognitive interference that occurs when processing one feature of a stimulus influences the simultaneous processing of another attribute of the same stimulus [13,14]. The task requires naming the color in which the word is written, not the word itself denoting a color. This is a relatively complex task that requires the simultaneous involvement and work of both hemispheres of the brain, and in the case of this test, the so-called "hemisphere conflict" can also be examined. Physiologically, the left hemisphere is dominant in most people, which is why they relatively easily name the word denoting a color, but experience difficulty when they have to name the color in which the corresponding word is written. This leads to the use of greater mental effort, a redistribution of tone from the left to the right hemisphere, and in some cases a person may increase their level of stress. When mental effort increases, the temperature emitted by the prefrontal areas of the cerebral cortex usually rises as well. The requirement for increased mental activity can also be demonstrated through visualization, using the appropriate IR sensor or thermal camera with precise beam targeting of the prefrontal areas.

2.4. Arithmetical Test

The second test is related to mental arithmetic, with the task being to subtract numbers mentally. In this way, activation is achieved predominantly in the left hemisphere (the left prefrontal area of the cerebral cortex) in subjects within the normal range. Performing this task may lead to a slight increase in certain values of cardiac activity or galvanic skin conductance, which can be associated with the individual psychological characteristics of the subjects being studied. Mental arithmetic is also used in the educational sphere [15].

2.5. Calculations and Data Processing

In order to check for dependencies between the individual measured parameters, we calculated the correlation coefficients between the uniform measured parameters for each participant, for the entire study, with accuracy to the second decimal place.

The anonymized data records have been further processed in MATLAB R2011b environment. Initially the data has been separated in five epochs spread totally in 14 min, discriminating three Rests (Rest 1 & Rest 3, both of 3 min each and Rest 2 – with 2 min duration) and two cognitive loads – Cog. Loads (Cog. Load 1 & Cog. Load 2, both of 3 min. each). Further, data was processed using bandpass Butterworth 12 dB/Oct digital filter with 0.5-70 Hz bandwidth and zero-phase shift. A followed processing with Chebyshev notch at 50Hz +/-5Hz with 18 dB/Oct) was also added for supply network hums eliminating. According to [16,17] bandpass filtering has been organized for the theta (4-7 Hz) & alpha (8-13 Hz) rhythms [18] with the same bandpass Butterworth 12 dB/Oct digital filter. The obtained results were used for averaged Alpha entropy H’ calculation, taking the Shannon approximation [19]. Other measures, like approximated entropy [16] and Kolmogorov’s entropy [20] have been also tested but showed computational & time resources higher demands in comparison with the selected one for the presented experiment duration. Results were taken for both epochs types (Rests vs Cog. Loads) and EEG leads.

2.6. EEG

EEG is a non-invasive method for recording an electrophysiological brain activity. Using a system of electrodes mounted externally on the scalp, non-invasive brain-computer interfaces based on electroencephalography (EEG) allow direct communication between the brain and external devices by tracking and recording electrical brain activity [21].

2.7. Galvanic Skin Response (GSR)

The Galvanic Skin Response (GSR) signal is that is primarily used for measuring of emotional arousal and physiological reactions, often applied in psychology, biofeedback, usability testing, and stress research. In the present study, GSR was recorded, using Mind-Reflection device of VERIM© & VERIM LAB software environment, using elastic band with two electrodes from the front phalanges of the middle & ring fingers of the non-dominated subjects’ hands, while staying in a calm, resting position. As some subjects have more dried fingers, additional Ten20 paste was used in order to achieve lower montage resistance. The results were stored in ASCII format file and further processed (see Section 3) [22,23].

2.8. Infrared Thermography

Infrared thermography is a remote method for detecting and tracking thermal radiation. It can be used to measure and study changes in surface temperature from a distance. Thermography is based on radiometry in the infrared spectrum. Thermal energy is electromagnetic in nature, with the distribution of energy described by Planck's law of radiation from an ideal black body. The following equation shows a mathematical expression of this law, according to which any body that has a temperature other than absolute zero emits radiation constantly, with the spectrum depending on its surface temperature [24].

$$L\left(\lambda ,T\right)={\displaystyle \frac{2h{c}^2}{{\lambda }^5}}{\displaystyle \frac{1}{exp\left(\frac{hc}{\lambda kT}\right)-1}},[\mathrm{W}{\mathrm{m}}^{-2}{\mathrm{Hz}}^{-1}{\mathrm{sr}}^{-1}]$$

(1)

where:

T represents the absolute temperature, h is Planck’s constant (6.626×10⁻³⁴ J⋅s), c denotes the speed of light (3.0×10⁸ m/s), and k indicates Boltzmann’s constant (1.381×10⁻²³ J/K), $ԑ$ denotes emissivity, $\sigma =5.67037.{10}^{-8}W{m}^{-2}{K}^{-4}$ denotes Stefan–Boltzmann constant.

Real bodies have different structures, different surface temperatures, and therefore emit differently. In order to be able to compare them and make different calculations related to their energy, the emissivity coefficient has been introduced. It is an individual parameter and characterizes the emissivity of each real body. The following equation shows a mathematical formula for its calculation.

$$\epsilon \left(\lambda ,T\right)={\displaystyle \frac{M_{\lambda }\left(\lambda ,T\right)}{M_{\lambda ,Blackbody}\left(\lambda ,T\right)}},$$

(2)

where:

$M_{\lambda }\left(\lambda ,T\right)$ is the spectral radiative power output [W·$m^{-2}$·μ$m^{-1}$], $M_{\lambda ,Blackbody}\left(\lambda ,T\right)$ is the ideal black body spectral radiative power output [W·$m^{-2}$·μ$m^{-1}$].

The normal body temperature of humans is about 37°C, making the operating range between 8µm and 14µm suitable for measuring such surface temperature.

2.9. Pulse Oximetry

A fast and accurate non-invasive technique for measuring and monitoring two vital physiological indicators during cognitive task performance - oxygen saturation (SpO₂) and heart rate by using Beurer PO 30 device. Their values ​​are related to cardiovascular activity. It is known that with increasing mental load, the consumption of carbon in the brain increases [25,26,27]. Studies have been conducted on the change in heart rate levels when solving mental arithmetic problems [28].

2.10. Setup

The experimental setup is a multimodal EEG-based non-invasive BCI system with integrated parallel measured indicators: heart rate (HR), galvanic skin response (GSR), facial temperature and oxygen saturation (SpO₂). The experiments have been performed in a normal office, quiet environment, while subjects were comfortably sitting in front of a computer screen, interfering with the experimental setup via desktop computer having a keyboard and a mouse interface in the feedforward direction and visual feedback via the setup monitor tests results’ responses (see Figure 1). Ten healthy volunteers have been studied (7 male & 3 female, average age 43.8 years, all right-handed) after signing an informed consent. The thermal camera is orthogonally aligned to the facial plane, positioned about one meter from the participant, at a height slightly above eye level, with its angular displacement not exceeding ±15° in order to avoid errors related to emissivity. To avoid inaccuracy in the thermal camera data, the region of interest (ROI) of the thermal image is localized in the same field for all participants, and in order to average the sensor noise, it includes at least 5×5 pixels. In order to avoid introducing artifacts and noise into the thermal and electrodermal signals, the light and temperature in the room are controlled and constant. The pulse oximeter is placed on the tip of the index finger of the left hand. The galvanic skin response (GSR) sensor is placed on the ring finger and little finger of the participant’s left hand. The dominant right hand remains free for task-related activities. All sensors are connected to a central laptop that manages data collection and synchronization in real time using a shared system clock. Thermographic images are captured with a FLIR E40 thermal camera and processed using FLIR Tools [29]. The brain activity has been recorded from the subjects’ scalp with a BrainBit Flex8 EEG wearable device within F3, F4, C3, C4, P3, P4 & O1, O2 leads (according to international 10-20 Jasper system, with sampling frequency – fs = 250 Hz), using spring-loaded gold-plated dry electrodes set and a supportive elastic cap with reference electrodes as ear clips and ground handled internally. Before each recording the electrodes impedance within the studied subject scalp was measured, assuring good contact for all leads and high amplitude of the recorded signals. All measuring and recording activities have been organized in BrainBit NeuroREC 3.0 environment installed on HP Victus Gaming Laptop 16-s0000nu platform (having the following key parameters: 32 GB RAM, AMD Ryzen™ 7 processor, 1TB SSD, NVIDIA® GeForce RTX™ 4060 video card & Windows 11 operation system) and the results have been saved in a raw form in EDF format for further processing. Figure 1 shows a participant photographed from behind and the experimental setup of the study.

3. Results

3.1. Facial Surface Temperature

3.1.1. The minimum correlation coefficient between the measured values ​​of the surface temperature of the face of all participants for the entire study period is – 0,63.

3.1.2. The average correlation coefficient between the measured values ​​of the surface temperature of the face of all participants for the entire study period is - 0,01.

3.1.3. The maximum correlation coefficient between the measured values ​​of the surface temperature of the face of all participants for the entire study period is 0,83.

Figure 2 shows the values ​​of the facial surface temperature of a large proportion of the participants for the entire period of the study.

The surface temperature of the face follows a behavior that is partially repeated as a type in different participants. It is clearly seen that at the end of the first break the temperature decreased in most measurements. Peaks are also seen during mental workload during both tests (between the 7th and 13th measurements and between the 17th and 23rd). There are exceptions, of course, but there are noticeable trends for facial temperature to increase during mental exertion and decrease during rest. It can be seen that the speed of these temperature variations is different for different participants.

3.2. Heart Rate

3.2.1. The minimum correlation coefficient between the measured values ​​of the heart rate of all participants for the entire study period is - 0,58.

3.2.2. The average correlation coefficient between the measured values ​​of the heart rate of all participants for the entire study period is 0,38.

3.2.3. The maximum correlation coefficient between the measured values ​​of the heart rate of all participants for the entire study period is 0,88.

Figure 3 shows the heart rate values ​​of almost all participants for the entire study period.

This measurement also clearly shows the increase in heart rate during the tests that require mental activity and the correspondingly lower values ​​during the breaks. Again, there are differences, which are mainly visualized as different peak heights in the different participants. Measurement number 7 is the last of the first rest. Then, during the first test, peaks in the heart rate of some of the participants are clearly visible (measurements with numbers 8, 9, 12 and 13). Then, during the second rest, drops in the heart rate levels of most participants are clearly visible (measurements with numbers 14, 15 and 16). The subsequent measurements with numbers from 18 to 23, which are during the second test, respectively this is the zone of the second mental load, again clearly show the increase in the heart rate levels of the participants and are visible on the diagram as peaks of different heights. At the end of the experiment during the last rest (measurements with numbers from 24 to 29) the diagram clearly shows the decrease in the heart rate values, which corresponds to the rest state. At the end of the experiment, the heart rate values ​​fall to the relative values ​​of heart rate at rest.

3.3. Oxygen Saturation

3.3.1. The minimum correlation coefficient between the measured values ​​of oxygen saturation of all participants for the entire study period is - 0,74

3.3.2. The average correlation coefficient between the measured values ​​of oxygen saturation of all participants for the entire study period is 0,02

3.3.3 The maximum correlation coefficient between the measured values ​​of oxygen saturation of all participants for the entire study period is 0,77.

Figure 4 shows the oxygen saturation values ​​of most participants for the entire study period.

Oxygen saturation measurements show relatively stable values. Some of the diagrams show small drops during periods of mental stress, but the stable behavior of this value indicates its low informativeness in this type of research.

3.4. Galvanic Skin Response

3.4.1. The minimum correlation coefficient between the measured values ​​of the galvanic skin response of all participants for the entire study period is – 0,58.

3.4.2. The average correlation coefficient between the measured values ​​of the galvanic skin response of all participants for the entire study period is 0,26.

3.4.3. The maximum correlation coefficient between the measured values ​​of the galvanic skin response of all participants for the entire study period is 0,90.

Figure 5 visualizes the values ​​of the galvanic skin response of all study participants.

The following Figure 6. shows the full combination of correlation coefficients of each individual measured parameter except EEG between each possible pair of participants without repetitions.

3.5. Electroencephalography (EEG)

Additional correlation coefficients r* have been calculated [16] for both epochs, regarding the E’ entropy. The aggregated findings, concerning these analyses are depicted in Figure 7 – Figure 8, as follows:

As it is clear from the averaged entropy H’ results for both Rests & Cog. Loads, the Rests episodes are somewhat oscillating with all brain areas, giving higher values after each cognitive load episode. These dynamic changes are clearly observed for both parietal & occipital brain areas, whilst giving slight domination to the left hemisphere leads (P3, O1) vs right ones (P4, O2). Lowest results are observed with central – right leads – C4, as far as, both frontal and central left ones are showing higher entropy values.

The situation, concerning both Cog. Loads’ episodes (Load 1 & Load 2) are giving almost equal dynamics for all recording leads, whilst giving some dominance to the left hemisphere (C3, P3 & O1 leads) in comparison to the right ones for the Load 2 episode. It should be also noted here that the discovered differences are not so high (in the interval of 0.2 - 0.8 from the overall H' entropy dynamics).

As for the correlations’ r* dynamics, the aggregated results demonstrate a frontal dominant correlation, concerning the fronto-central zones during Cognitive loads identification. In comparison the parietal correlations though still significant, are giving lower results for the right hemisphere. Similar results are also observed and in the occipital zones. Regarding the Rests’ parts of the experiments – the results are more dynamic, giving also aggregated and synchronized response to the frontal, but left hemisphere (for F3 leads), whilst the right ones of the frontal and central zones demonstrate lower correlations (for leads F4 & C4), adding also same response for C3 leads. Apart of these findings, lowest correlations are also observed with the left parietal leads of P3 and partially occipital ones with O1. The rest of the responses from P4 & O2 are giving much better correlation performance. Most significant results are observed between first two Rests, and less – between second and third ones.

Obviously, the outlined findings do not demonstrate a convincing enough and clear discrimination between Rests & Cog. Loads episodes of the experiment as the EEG signal has much more complex nature [18], that could be studies also deeper with the emotional context [30]. So, further complexity increasing has been done, adding the theta frequencies band. Following the ideas of [31], the averaged alpha/theta ratios - ρ’ α/θ have been calculated for both Rests & Cog. Loads episodes of the experiments (see Figure 8).

Figure 9. Averaged ρ’ - alpha/theta ratio for all studied subjects, spread amongst the eight EEG leads for Rests – Rest 1, Rest 2, Rest 3 & Cog. Loads – Cog. Load 1 & Cog. Load 2 epochs of the experiment.

Figure 9. Averaged ρ’ - alpha/theta ratio for all studied subjects, spread amongst the eight EEG leads for Rests – Rest 1, Rest 2, Rest 3 & Cog. Loads – Cog. Load 1 & Cog. Load 2 epochs of the experiment.

This time the obtained results have shown a clear discrimination capability within all EEG leads and experiment Rests & Cog. Loads episodes, providing almost doubled averaged ratios - ρ’ between them.

Finally, it should be noted that the presented findings here could be speculatively considered as a natural brain memory usage increase during cognitive tasks solving, i.e. Loads episodes (that could be referred to increased EEG theta band) in comparison to the Rest states, mostly referred to increased EEG alpha band. The averaged entropy H’ dynamics also is possible to be explained in this sense, as far as the frequency changes towards higher levels are also related to increased entropy during Loads episodes. As the implemented frequency bands of alpha & theta are just part of the EEG spectrum the rest of beta & gamma could also be used but for deeper exploration of the origins of the brain dynamics changes, whilst here just a simple & reliable metrics have been found from a general measurement perspective.

4. Discussion

The analysis of the collected physiological data (facial temperature, heart rate, oxygen saturation, and galvanic skin response) across ten participants reveals distinct patterns and inter-individual variability in response to cognitive load. The following observations, derived from the dataset, complement the EEG findings and provide a broader understanding of the physiological correlates of mental workload.

1. Galvanic Skin Response (GSR) Exhibits the Most Pronounced and Consistent Increase During Cognitive Load

The GSR data shows the most significant amplitude changes between rest and task periods across the majority of participants. Values frequently increased by 100-300% during the Stroop and Arithmetic tests compared to baseline rest periods.

This is reflected in the correlation analysis, where GSR showed the highest maximum correlation coefficient (0.90) between participants, indicating that for some individuals, the phasic response pattern was very similar. However, the wide range (minimum: -0.58, average: 0.26) also highlights significant inter-subject variability in baseline skin conductance and response magnitude.

Observation: GSR serves as a highly sensitive, though individually variable, indicator of autonomic nervous system arousal during cognitive tasks, with a clear and strong response to increased mental demand.

2. Heart Rate (HR) Shows a Reliable Increase During Cognitive Tasks

A clear and consistent trend of increased heart rate was observed during both cognitive tasks (Stroop and Arithmetic) compared to the rest periods. The data shows an average increase of approximately 5-15 BPM from resting baseline to the peak of cognitive effort.

The correlation coefficients for HR between participants were positive on average (0.38), with a maximum of 0.88. This suggests a more uniform directional response (increase during load) across the cohort compared to other measures like temperature or SpO₂.

Observation: Heart rate provides a robust and generally consistent metric for distinguishing states of mental strain from rest, aligning with the expected cardiovascular response to cognitive stress.

3. Facial Surface Temperature Demonstrates Complex and Biphasic Patterns

Contrary to a simple uniform increase, facial temperature dynamics were more complex. A common pattern observed in multiple participants was an initial decrease in temperature at the onset of a cognitive task, followed by a subsequent rise as the task continued.

The correlation between participants was the lowest on average (-0.01), with a wide range from -0.63 to 0.83. This indicates that while some participants showed strong correlated patterns (e.g., a clear temperature rise with load), others showed inverse or uncorrelated responses, making it a less reliable standalone metric without individual baselining.

Observation: Facial thermography captures dynamic physiological changes, but its response is highly individualized. The biphasic pattern (initial drop followed by an increase) may reflect rapid vasoconstriction followed by increased cerebral blood flow and metabolic heat.

4. Oxygen Saturation (SpO₂) Shows Minor Fluctuations with Low Inter-Participant Consistency

Oxygen saturation levels remained within a normal physiological range (95-99%) for all participants throughout the experiment. While subtle decreases (1-2%) were sometimes recorded during cognitive tasks, these changes were not consistent across all subjects or all task periods.

This is supported by the correlation data, which shows an average correlation near zero (0.02) between participants, indicating no consistent, synchronized pattern of SpO₂ change in response to the cognitive loads in this study.

Observation: Peripheral oxygen saturation, as measured by pulse oximetry, appears to be a stable parameter largely unaffected by the short-term cognitive workload induced in this protocol, and is not a primary discriminator between rest and task states in this context.

5. High Inter-Participant Variability Underscores the Need for Personalized Assessment

The calculated correlation matrices for all non-EEG parameters consistently show a wide spread of correlation coefficients (from strongly negative to strongly positive) for every possible pair of participants.

This finding strongly suggests that while the direction of change for parameters like HR and GSR is often consistent (e.g., HR generally goes up), the specific pattern, magnitude, and timing of each individual's physiological response is unique.

Observation: The significant inter-participant variability across all measured physiological channels highlights the challenge of creating a one-size-fits-all model for cognitive state assessment and reinforces the potential value of personalized baselines and calibration for BCI applications.

Summary of principal findings

Among the autonomic measures, heart rate (HR) demonstrated the clearest sensitivity to task demands: it increased during both cognitive loads and was more pronounced during the more difficult arithmetic task. Galvanic skin response (GSR) also contained information related to the task, but raw data showed large between-subject variability; after normalizing for each subject and centering to baseline, GSR effectively differentiated the higher load condition. Facial temperature did not show sharp, task-related peaks; instead, it experienced a gradual upward drift over time, likely due to vasomotor activity and thermal inertia. SpO₂ remained relatively stable, as expected during seated cognitive tasks, thus serving as a physiological baseline. EEG was summarized by alpha/theta ratios across eight leads per epoch; this central measure is theoretically expected to decrease with increased workload, providing a complementary perspective to the peripheral measures.

Fast versus slow channels

The data correspond with established time scales. Fast changes include HR (beat-to-beat and short windows), phasic GSR (event-related peaks), and EEG alpha/theta, which can fluctuate within seconds of task start. Intermediate changes involve tonic GSR, showing gradual drifts over minutes. Slow variations are seen in facial temperature, which accumulates over minutes rather than reacting immediately to task events. Understanding these time scales is essential for selecting appropriate windowing methods—using event-locked windows for HR, GSR-phasic, and EEG, and epoch averages and trends for temperature.

On integrating heterogeneous modalities

Multimodal integration is valuable but should be used sparingly. No single channel suffices for all participants and tasks; combining central (EEG) with peripheral (HR, GSR, temperature) signals improves robustness and supports convergent validity (e.g., HR increase and GSR-phasic increase alongside EEG α/θ decrease during load). However, integration should happen in stages:

1. Normalize each subject individually (e.g., z-score or min–max); for GSR, separate phasic from tonic components and normalize to Rest 1.

2. Extract compact epoch-level features per modality (e.g., HR mean and HRV; GSR peak rate/amplitude and tonic level; facial temperature mean and change from previous rest; EEG α/θ per region).

3. Perform late fusion (weighted z-sum or a small multivariate model) after validating each modality independently.

4. Conduct cross-modal checks (repeated-measures correlations, cross-correlation for lead–lag).

Mapping to the prior hypotheses:

• H1a (Stroop vs Rest1): HR tends to increase; evidence is weaker than for the arithmetic task, indicating a smaller load. GSR shows limited evidence before normalization; after normalization, effects are more prominent in the higher load condition. Temperature and SpO₂ do not show task-specific differences.

• H1b (Subtraction vs Rest2): HR strongly supports this; normalized GSR also differentiates this contrast; temperature does not; SpO₂ remains stable.

• H2 (Subtraction vs Stroop): HR distinguishes task difficulty; normalized GSR indicates a stronger response to the more challenging task; temperature and SpO₂ do not.

• H3 (Rest3 vs Rest1): HR and GSR generally return to baseline (no consistent difference), whereas facial temperature is slightly higher later in the session, consistent with slow changes accumulatio.

Role of EEG alpha/theta

The averaged alpha/theta ratios were calculated across eight leads per epoch to assess cortical states. Typically, α/θ ratios decrease with increased workload. In this dataset, the alpha/theta metric serves two complementary purposes: (i) Confirmatory, by testing whether α/θ (Rest) is greater than α/θ (Stroop) and α/θ (Rest2) exceeds α/θ (Subtraction), preferably using a within-subject mixed model; and(ii) Convergent, by examining correlations between α/θ and HR and GSR across epochs, expecting negative correlations with HR and phasic GSR during task load. Even if power is limited at [missing text], directional trends enhance the interpretation of results across multiple modalities.

Correlation structure and lead–lag

Simple epoch-wise correlations are useful but should be supplemented with repeated-measures correlation to prevent inflation caused by between-subject differences and lagged analyses, such as cross-correlation, to investigate if EEG changes occur before autonomic responses. With n=10, such timing assessments should stay descriptive but can inform future study designs, like using shorter windows around task onsets.

Limitations

The sample size is small (ten participants), which limits the ability to detect small effects and restricts the complexity of multivariate models. Epoch lengths are inconsistent, and the slow dynamics of temperature can obscure task contrasts. Large between-subject variability in GSR requires normalization. These limitations led to the use of mixed models, conservative feature sets, per-subject scaling, and planned contrasts.

Implications and future work

The results propose a practical approach for non-invasive workload assessment: use HR as a quick, reliable marker; include normalized phasic GSR to detect higher loads; utilize EEG α/θ for central validation; consider facial temperature as a slow-changing context indicator; and keep SpO₂ as a measure of stability. Future research should involve larger samples, standardize event-locked windows, and pre-register concise feature sets. With increased data, it becomes possible to analyze a latent workload factor that combines EEG and autonomic features using Bayesian or SEM methods, and to evaluate how well the model generalizes through cross-validated predictions.

In summary, multimodal sensing is justified even at small scales when each modality is processed on its natural time scale, normalized per subject, and interpreted within a confirmatory mixed-model framework. HR and normalized GSR consistently indicate load (and task difficulty), EEG α/θ provides central convergence, facial temperature reflects slow drift, and SpO₂ confirms physiological stability—together supporting the viability of a non-invasive, multimodal approach to current psychological state assessment.

5. Conclusions

The correlations found between the individual measured parameters, the calculated correlation coefficients between the different participants for the same modality and the results presented in the article could be considered as an indicator of the capabilities of non-invasive multimodal BCI systems. The results suggest that they could be used to assess mental status. Understanding these factors can help to better cope with everyday challenges and improve the overall quality of life. Stress and mental state can lead to physical illnesses such as depression and reduce an individual's concentration and productivity, making it challenging to find new ways to prevent the effects of stressful situations. The results of the study can be used in the future as a basis for other more specialized studies of psychological stress.

In order to build successful practices for studying the influence of cognitive stress, as well as methodologies for overcoming stressful conditions, taking into account the difficulty of such studies and the many influencing factors, we would like to point out the need for additional research.