Emotional State Measurement Trial (EMOPROEXE):a Protocol for Promoting Exercise in Adult and Children with Cerebral Palsy

1. Introduction

The objective of this experimental protocol is to determine the relationship between physiological measurements and emotional state in children and adults with cerebral palsy. Our goal is to collect objective information on the state of the users in order to achieve greater effectiveness in rehabilitation therapies and in the promotion of physical activity. The knowledge of the user emotional state is important to conduct the appropriate activities for their rehabilitation.

The experimentation will be conducted with a group of users belonging to the Association of People with Cerebral Palsy of Seville (ASPACE) and the Special Education School of Seville Director Mercedes Sanroma.

The methodology will be applied to groups of users of different ages with different capacities. For this reason, this is a flexible methodology that proposes different tasks according to the context. Some of the tasks involve physical activity, which must be taken into account due to the possible disturbance that it can introduce in the measures taken.

Having an emotional characterization of the user based on their physiological measurements will allow us to do a real-time evaluation of the feelings of users during the performance of the tasks. In this way the activity can be conducted better by adapting the necessary parameters so that it can be calculated as efficiently as possible and lead to better results.

The challenge we propose to address is that the subjects have difficulties in expressing and recognizing emotional states, which rules out the use of self-assessment tests to contrast and compare with the measures taken. This makes us resort to their caregivers or relatives or, alternatively or complementary, take measures in contexts or situations of daily life where the emotional state induced in the subject is known.

This experimentation is part of a subproject called AAI (Augmentative Affective Interface), which belongs to the AIR4DP (Artificial Intelligence and Robotic Assistive Technology devices for Disabled People) coordinated project that is funded by the Ministry of Science, Innovation and Universities (I+D+i Projects 2019 - Society Challenges). The main expected result of the AIR4DP project is the implementation of assistive technology that allows incorporating the latest advances in artificial intelligence (AI) to improve the quality of life of people with disabilities. The AAI subproject seeks an improvement in user interaction with systems achieved through knowledge of the emotional state based on physiological measurements and image. If this knowledge is injected into the robotic platform or into the software used in rehabilitation therapies, a greater immersion and motivation of the subject in the proposed activities will be achieved.

2. Review of the state of the art

In order to have a knowledge base, three searches have been carried out with the following keywords:

• Emotion & detection & physical & activity. These keywords have been used with the aim of exploring methodologies that allow detecting the emotional state during physical activities, or the proposals of physical activity according to a specific emotional state.
• Emotion & detection &cerebral & palsy. With this search, we intended to detect the studies related to the determination of the emotional state in personnel with cerebral palsy. The special characteristics of this population mean that the usual methodologies are not fully applicable, so it is of interest to study cases where this type of measurement has been made.
• Emotion & elicitation & music. Music provokes emotions in subjects. The qualities of sound: Frequency, timbre, duration and intensity influence the induced emotions, hence its use in therapies. It can be a way of bringing the subject to the desired emotional state to correlate parameters measured in it or a form of motivation to carry out activities.

In [1] a health assistant system for subjects with depression is proposed. The system has three facets: the first remembers the taking of medications, the second measures the emotional state of the subjects focuses mainly in electroencephalography (EEG), the third proposes exercises based on the previous determination of the emotional state. In [2] the purpose is to detect the state of relaxation based on physiological signals, specifically in the galvanic skin response (GSR), also called electrodermal activity (EDA). In [3] an experiment is carried out to measure stress in real life, where 16 subjects participate. A smartwatch measures temperature, pulse, GSR and acceleration data. Contextual weather information is also measured. The incorporation of the environment measurement increases the precision in the detection of the stress state. In addition, these measures are based on responses to stress self-assessment questionnaires. In [4] a system is proposed to motivate and monitor the physical activity of the elderly, the system is based on the emotional state for activity proposals. They rely on the database called AMIGOS [5]. The questionnaires used for both users and caregivers related to their experience using the system are shown: the questionnaire for users is shown in Table 1 and that of caregivers in Table 2.

In [6] a robotic platform is proposed to encourage physical activity in older people, their emotional state is measured through images. In [7] cognitive fatigue is measured using a mobile application and 18 people participate in the experiment. The measure is based on the responses of the subjects to mobile games, a questionnaire and facial recognition with affective. In [8] different methods of classification and processing of the EEG signal are evaluated to detect emotional states caused by a game.

There are few works dedicated to the evaluation of emotional states of people with cerebral palsy (CP). This topic has constituted the second of the searches carried out to determine the state of the art. In [9] visual evoked potentials using IAPS imaging are used to analyze evoked emotions in children with CP versus typically developing children. The study is supported by questionnaires in the family environment (KIDSCREEN52) and also the Self-Assessment Manikin to the participants who were chosen with the appropriate cognitive level to be able to complete this test. The test is done with 15 children with CP and 14 children with normal development. It is observed that images with affective content induce less amplitudes in brain responses in children with CP than in those with normal development, which seems to indicate a lower ability to detect emotions. These emotion elicitation techniques will not always be valid, it is something to study with the help of the professionals who attend to the subjects.

In [10], a study is done with a 10-year-old girl with a GMFCS (Gross Motor Function Classification System) of III. Virtual reality is used. Their emotions are analyzed based on facial gestures. In [11], a case study of a girl with cerebral palsy with many communication difficulties with the caregiver is presented, from an image recognition application in PYTHON they are able to detect patterns that inform the caregiver of their status.

Some works in which music is used to induce emotions in the subjects are described below and it is analyzed that it will arouse different emotions depending on the musical properties of the piece. In [12] emotions are detected while the subject plays a racing video game, the EEG signal is chosen and the classifier is developed by taking the signals from a specific database of emotions. The idea is to insert musical segments into the game according to the emotions detected in the player. In [13] an experiment is carried out 13 bars in La major and minor are used as stimuli, different self-assessment scales are used to assess the musical pieces and the ECG (Electrocardiogram) signal and respiration are measured. It is observed that the mode significantly influences the physiological signals. In [26] an experiment is carried out to see how music affects brain activity. Two types of stimuli are used: generated music and classical music. The music generated is used so that a possible familiarity with it does not affect the emotional state, which is why it is based on music that the participant will not know from the outset. Classical music is used in search of strong affective responses. EEG and FMRI (Functional Magnetic Resonance Imaging) measurements are taken and it is observed that the musical stimulus affects the asymmetry in the responses of the two cerebral lobes. In addition to physiological measurements, users must also respond to tests on their emotional state. In [15] it is intended to elicit astonishment in users and for this, different music stimuli and Virtual Reality (VR) are combined, the results are obtained through various self-assessment tests. Physiological measurements are not performed. In addition, as the elicitation of amazement also depends on the personality of the subjects, two additional scales are used, the first measures the predisposition to experience positive emotions and the second musical preferences. In [17] an experiment is carried out with 12 people in order to find out the relationship between the EEG signal and emotions. The asymmetry in the value of the EEG signal and its relationship with emotions are studied. To elicit emotions, images from the FACS (Facial Action Coding System) are used, in order to elicit them by imitation and these images are accompanied by music. In [18] the objective is to elicit emotions in the subjects for which movie clips have been used, the clips are chosen to elicit emotional states of happiness, sadness, fear and relaxation. To modify the emotion induced by the scene, music and the color of the ambient light are used for two minutes.

3. Materials and Methods

3.1. Design of the study

The first part of the study will be descriptive and observational where the choice of those parameters (dependent variables) that adequately characterize the emotional state of the subjects will be sought. The second part of the study will be analytical where the influence of music on changes in the selected dependent variables will be determined. In this second part, a single case study will be made, type AB, for each subject, where, in condition A, the rehabilitation exercise is carried out in silence, while, in condition B, it is accompanied by motivating music.

3.2. Participants

The participants, as already indicated, belong to the ASPACE association (adults) and the director Mercedes Sanromá special education school (children). The study will involve 40 subjects: 20 adults and 20 children. The following criteria are proposed for this group of participants.

3.2.1. Inclusion criteria

• People with a recognized disability, caused by a disease or permanent health situation.
• Be between 2 and 65 years of age.
• Have a degree of functional ability in mobility domain moderate-low (measured through items related to their motor functionalities of the International Classification of Functioning, Disability and Health (ICF) [16] in the case of adult population and Gross Motor Function Classification System (GMFCS), [19] and Manual Ability Classification System (MACS) [20] in the case of children). In [21] a study of this population was conducted and the two scales were homogenized to measure adults and children in the same way.
• People with motivation to use technologies and/or who can use wearable devices during the intervention time.
• People who come weekly to the collaborating centers.
• People without hearing impairments.

3.2.2. Exclusion criteria

• Presenting a health situation that is incompatible with the use of technology ((e.g. use of respirator, pacemaker, sensitive skin...).
• Have a very limited cognitive capacity, which prevents you from following the instructions for the proper use of assistive technology (measured through items related with this in the ICF scale in adults and Communication Function Classification System (CFCS) [22] in children).
• Not having adequate human support.
• People with hearing impairments.

3.2.3. Recruitment of participants

All people who meet the inclusion criteria will be invited to participate in the study. The recruitment of participants will be carried out by the TAIS (Technology for Assistance Integration and Health) research group of the University of Seville, through contact with the collaborating centers. In all cases, a cover letter will be delivered to the managers or directors of the centers with all the information about the project. The participants and/or their representatives will sign the informed consent form.

3.3. Instruments of measurement

3.3.1. Tests and questionnaires

Tests to determine the basic state or those to be used before and after the intervention of the participants. Questionnaires of the basic state of the subjects. It is important to know the quality of life and emotional situation of the subjects to be measured, with the aim of subsequently contrasting the physiological measurements taken. There are two populations, children and adults. In addition, the cognitive status of the subjects may vary. All of these are factors to be taken into account when applying the measurement instruments. Sometimes, it will have to be the family or the professionals of the centers responsible for completing the questionnaires.

• ICF for the adult population [16].
• MACS [20]for children.
• GMFCS) [19] for children.
• CFCS [22] for childrem.
• KIDSCREEN Questionnaire 1 will be used in its 10-item version for the the evaluation of the child population; it is an instrument that measures the quality of life related to health.
• Musical Preferences Questionnaire. This is about asking about songs that motivate the subjects and generate a positive and active emotion.
• EVEA scale and free text to be filled in by caregivers or relatives. The EVEA scale, according to [18], is consistent and has the ability to detect changes in mood. This scale will be passed at the beginning of the data recording once the sensors have been placed and at the end of recording time.

3.3.2. Devices for recording physiological data

During the sessions, data provided by the wearable devices developed in the context of this project will be recorded. Figure 1 shows a diagram of the sensors used.

These are 4 devices that distribute sensors throughout the body: 4 inertial units on the wrist, ankle, chest and head; a wrist temperature sensor; a sensor of electrical activity of the skin (EDA) in the phalanges (it is measured by dry electrodes placed on the hearten and hypothenar eminences of the dominant hand); an electrocardiography sensor on the chest and 8 channels of electroencephalography (EEG). In addition, the ambient temperature can be recorded.

One of the wearables is the OpenBCI device 2. It is an 16-channel bioamplifier for EEG measurement, it measures at a maximum rate of 250 Hz and includes an inertial unit that measures at 25 Hz. In Figure 2 electrode position is indicated.

It will be analyzed the following independent variables:

• Average kinetic energy measurements (in joules) using inertial sensors. They provide information about the energy expenditure that these entail.
• Instantaneous Heart Rate (HR), in seconds. A wearable placed in the chest with Ag/AgCl electrodes for ECG is used. The position of R wave is determined using an appropriate algorithm and then time difference between two consecutive R waves is calculated, this time difference is used to calculed HR.
• The ratio between low frequency, (LF) and high frequency, (HF), (LF/HF) components of HRV (Heart Rate Variability) measured from ECG. The ratio shows the balance between the SNS (Sympathetic Nervous System) and the PNS (Parasympathetic Nervous System).
• Temporal parameters of HRV. HVR can be measured using temporal parameters such as: SDNN Standard deviation of NN intervals; RMSSD Root mean square of successive differences between normal heartbeats; pNN50 Percentage of successive RR intervals that differ by more than 50 ms.
• Tonic Skin Conductance Level (SCL) This signal is the background tonic of EDA.
• Parameters of Phasic Skin Conductance Response (SCR). This signal are constituted by the rapid phase components of the EDA.
• Fractal dimension of EEG. The EEG signals are highly complex and dynamic in nature. Fractal dimension (FD) is emerging as a novel feature for computing its complexity. We will use the Higuchi’s algorithm.
• Spectral Entropy (SE) of EEG. SE can be used for computing EEG complexity. To do that, the power spectral density (PSD) must be obtained as a first step. After normalizing the PSD by the number of bins, which can be viewed as a probability density function conversion, the classical Shannon’s entropy for information systems is then calculated.
• EEG coherence. The interactions between neural systems, operating in each frequency band, are estimated by means of the EEG coherence. While neural synchronization influences EEG amplitude, the coherence between signals captured by one pair of electrodes refers to the consistence and stability of the signal amplitude and its phase. Two brain areas connected should show a signal delay in time domain that is measured as a phase shift in the frequency domain.

The data are recorded in a synchronized manner thanks to a software application (Figure 3) designed for this purpose [23].

3.3.3. Contexts and measurement frequencies

Four sessions will be held, divided into two parts:

Selection of dependent variables The aim of the first two sessions is to be able to count a reference level for physiological variables in activities that provoke pleasant and unpleasant emotions, so that they can be used as a reference in Part 2; the purpose is to try to avoid dependence on the EVEA tests since the subjects will not always be able to express their emotions. The EVEA test is used as a reinforcer for a possible automatic classifier.

• Session1: measurement of parameters when the subject is in a pleasurable activity of daily life in the center.
• Session 2: measurement of parameters when the subject is in a discomfortable activity of daily life in the center.
  These sessions will be determined by conversation with the caregiver since they are particular for each subject.

Half of the participants will start with session 2 and then do session 1, while the rest will follow the reverse order.

Effect of music on the dependent variables during the performance of rehabilitation exercises

• Session 3: Measurement of parameters to the subject during the performance of rehabilitation activities in the center.
• Session 4: Measurement of parameters to the subject during rehabilitation activities in the center. The session will be accompanied with music according to the preferences of the subject.

The pleasant motivational music to be played during session 4 will be selected by each user according to their musical preferences or, failing that, by their caregiver. The rehabilitation activity should be a light exercise for the user, such as pedaling, limb extension, or any other that is measurable through inertial units. The specific activity that each user will have to perform will be determined by the medical staff and/or physiotherapist of each center, as it will be limited by the movement capacity of each participant.

Although each session has a different theme, the structure of the sessions is similar. First, the sensors are placed on the volunteer. Once it has been verified that the data are collected in an adequate manner, data recording begins while the user is answering the EVEA test. This first part of the recording will be used as a baseline for the session, which should last at least two minutes. After that, the activity will start, which will not last more than 15 minutes; and to finish, a new EVEA test will be filled in, with identical restrictions to the first test. With these initial and final baselines, the differential of the measurements of each session can be detected, in addition to the analysis of the evolution of the subject during the activity. For each user, the protocol should be completed in two weeks, during the first week sessions 1 and 2, and during the second week sessions 3 and 4. Figure 4 shows a diagram of the described protocol.

4. Statistical methodology

4.1. The Sample size

The population of the centers is approximately 200 individuals (100 from each center, adults and children). According to Cochran’s formula for estimating the sample, assuming, in the worst case scenario, a value of p=q=0.5 (maximum variance), a confidence interval of 95% and a margin of error of 15%, the sample size would be of 36 individuals. Therefore, a sample size with 18 adults and 18 children seems adequate.

4.2. Data analysis

In general, numerical variables will be expressed as mean (M) and standard deviation (SD), including range, minimum and maximum.

For the first part of the study, comparisons will be made between the dependent variables obtained between sessions 1 and 2 for each subject using the permutation test of the difference in means between different data windows. Those dependent physiological variables that mark significant differences for most of the subjects will be chosen. Changes in EVEA test responses between sessions 1 and 2 will serve as reinforcement for detecting the level of emotional change.

The results of the EVEA test will also be correlated with the base state questionnaires, to determine the existence of some type of dependency between the base level of EVEA and its variation with respect to the base state. This dependence will be explored with the Spearman or Pearson correlation test.

For the second part, first, the variation in EVEA for sessions 3 and 4 will be analyzed to find out if there is a significant dependence on the independent variable (music) between stages A and B for all the participants. We will use the Kruskal-Wallis test, which does not require any type of assumption regarding the normality and homoscedasticity of the samples. These variations will also be correlated with the quality of life scale(ICF o GMFCS MACS CFCS KIDSCREEN), as was done with the data from the first part.

For the physiological variables and the average kinetic energy, it will be determined if their variations are significant between A and B by means of the Kruskal-Wallis test applied to the group of subjects, but also at the individual level by means of the permutation test.

The significance of the tests will be set at three levels indicated with *(p<0.05), ** (p<0.01) and ***(p<0.001).

5. Conclusions

People with CP have particular difficulties in recognizing both their mood and fatigue. This means that in rehabilitation routines or in the performance of physical activity, it is sometimes difficult to properly program the exercise so that they find it motivating and get the most out of the time devoted to such activity.

There are questionnaires that allow the measurement of both the emotional state [24] and the fatigue state [25], but not always a user with CP has the necessary cognitive and communication skills to rely on these questionnaires.

It is necessary to resort to more objective methods and the measurement of biosignals is a good alternative. But in order to rely on biosignals, it is necessary to study whether the changes generated in the biosignals are really significant. To do this, we must put the subjects in contexts where we can be sure what their mood is going to be, these contexts belong to activities that are part of their daily routines and of which we have experience of their response to them. The caregivers and the family environment should support the choice of these routines where the biosignals will be recorded. Whenever possible, these measures can be reinforced with tests to measure emotions and physical state that can be performed by the subjects or caregivers in case the subject does not have physical or cognitive abilities. On the other hand, signal processing should be performed to extract parameters that should then be studied statistically to assess their robustness in determining a state. It is also important to assess how the fact that the person is in motion affects the record. Appropriate techniques should be studied to eliminate the noise produced by this fact so that it does not affect the extraction of information about the condition.

In addition, we want to evaluate how music can be a motivational factor that improves the quality and time dedicated to physical exercise. Therefore, in the proposed experimental protocol, an analytical study of type AB is introduced in the second part. In the third part of the study of the state of the art, works are presented whose common denominator is that musical parameters can induce emotions and the measurement of the change in them is mainly done with the EEG signal. The proposed protocol will take into account these factors for the choice of music and will introduce the measurement of other types of additional physiological signals.