Impact of Varying User Characteristics on The Accuracy of Fall Prediction Algorithms Using Custom Wrist-Worn End Device

1. Introduction

According to WHO statistics [1], people are living longer, and the current population is expected to live well into their sixties. Current statistics (1st October 2022) state the following.

• The speed of ageing is increasing;
• In the year 2020, people aged more than sixty years outnumbered young kids under the age of five;
• By 2050, the population of geriatrics is expected to double from 12% to 22%. People over 60 are expected to be around 2.1 billion, and the number of people above 80 is expected to reach 426 million.
• Two-thirds of the ageing population is expected to be in the low- and middle-income range.

The ageing process causes a decrease in physical and mental capacity, other than biological variations, which are a natural part of ageing. With aging comes retirement and relocation. The elderly are generally considered frail and dependent and, hence, are shunned. Many of them are forced to live in a low-income retirement home. The changes in their health, financial and social conditions expose them to health risks; they especially become more prone to accidents, including falls.

United Nations has declared 2021 to 2030 as the “UN Decade of healthy ageing” and are supporting the use of technology to improve the quality of life of the elderly.

With recent advances in healthcare systems, specifically with IoT and medical applications integration, research in medical sensors and machine learning algorithms has been a considerable boost.

Machine learning and Deep Learning(DL) have widely investigated topics in the case of geriatric fall detection. To train the ML/DL algorithms, a large amount of data related to fall and non-fall activities is required. Several public datasets for fall are available. The public datasets vary in terms of (a) sensors used, (b) the number of volunteers, (c) demographics such as age, gender, and existing health conditions, (d) activities performed (e) data gathering techniques. Several researchers [2] have brought out the difficulty of comparing datasets. Therefore, it is also difficult to conclude why specific ML/DL algorithms perform better when compared to others. Most public datasets do not provide the details of data collection. The datasets used for fall detection have a set of ADLs and a set of fall data. Public datasets do not always list the ADL activities, the type of falls, how often these activities were performed and how long each activity lasted.

Two types of sensing methods are associated with fall detection (a) wearable and (b) ambient sensors. The classification is based on the placement of the sensor. The sensors can be placed on the body of the elderly or maybe placed in the environment around them. Body-worn sensors usually are accelerometers, gyroscopes, and biometric sensors. Environmental sensors include image, vibration, and audio sensors.

Several public datasets, such as Mobi Fall, K Fall, SiS Fall, and SmartFall, use body-worn sensors primarily using accelerometer-based sensors. This paper concentrates on data collection using wearable sensors, specifically IMU (Inertial Measurement Unit) sensors, gyroscopes, and magnetometers. Even with body-worn sensors, there is a considerable variation in the data collected. This is due to the positioning of the sensor on the body. The sensors maybe placed on the torso, thigh, or waist. Based on the placement of the sensors, the data collected. Moreover, the performance of the ML algorithms vary.

In recent work, the authors in [3] have given a study of recent trends in Human activity recognition (HAR) and human behaviour recognition (HBR) and how high-quality sensors like electrical, magnetic, mechanical (kinetic), optical, acoustic, etc., are being used for the same.

Raw sensor data is not directly fed to the ML algorithms; sensor data is processed to extract the relevant features; in the case of wearable sensors, these are usually statistical parameters – such as average, mean, maximum, minimum, standard deviation, kurtosis, skew, etc. Public datasets only have the extracted features available. Some have only raw accelerometer values; each public dataset produces different accuracies when applying ML algorithms. This makes it difficult to recommend a single ML algorithm for fall detection. Also, the accuracy of the ML algorithm cannot be coordinated with the data points as no information regarding the volunteers and their characteristics are available. Therefore, the following question remains to be answered while using public datasets. (A) What is the reason for varying accuracies for the same ML algorithms across different datasets (B) Are the volunteered demographics related to varying accuracies, and if so, how? (C) How much data is required to train the algorithms to obtain good accuracies, especially since DL algorithms require a large amount of data to converge (D) Can different datasets be used for training and testing, and how will the accuracies of the ML algorithms will be affected. (E) What will be the actual accuracy obtained when the system is used on the actual target users (i.e.) the elderly. The volunteers cannot be drawn from the geriatric populations due to the health risk involved with falling.

Our paper presents a wrist-worn end device that is part of a more extensive IoT system. The end device will collect data and execute the ML/DL algorithms to detect falls and alert healthcare professionals. The ML algorithms are run on the end device to eliminate network latency and connectivity-related issues, which are frequent in India; hence, fall alerts will never reach the health authorities if the ML algorithms are run on the cloud. Only long-term health monitoring and analysis will be done on the cloud. This architecture is termed Dew Computing in IoT. The architecture of the system used is shown in Figure 1.

This end-device is built around a powerful System on Chip (SoC) that is Qualcomm Snapdragon 820c [4]. The 820c chip has been developed specifically for wearable and IoT applications. We used MAX30102 [5] Heart rate and SP02 sensor, MPU6500 [6], which gives 3-axis acceleration, 3-axis linear acceleration and 3-axis gyroscope data and GY273 [7]Magnetometer chip for data collection. All the sensors are interfaced to the SoC via the I2C interface using a Mezzanine board.

This device needs to be trained with a large dataset so that we can run compressed ML/DL algorithms on it. The need for compressed algorithms is because running a full-scale algorithm will require a large amount of memory that is not available on SoCs.[8] shows the high latency incurred when running ML algorithms on SoCs. In order to train the system, we started with data collection, cleaning and analysis. This paper elaborates on the process we have used for data collection, cleaning, feature extraction and analysis of the data collected. Various ML algorithms were run on the data collected, and analysis was done on various user characteristics and their impact on the ML models. Some of the features considered were (a) Separate test and train data, (b) age, (c) gender (d) physical condition such as height, weight, and any pre-existing health condition. This paper provides the result of the analysis. Also, it addresses the issue of data collection and the impact of the data collected’s characteristics on the ML algorithms’ performance.

2. Background of Work

“Inadvertently coming to rest on the ground, floor, or other lower levels, excluding intentional change in position to rest in furniture, wall or objects”, is defined as a fall by WHO [1]. Falls can be detected using multiple mechanisms and methods. This section gives a brief review of the research done in terms of data collection and analysis done for fall detection in the elderly.

2.1. Sensors

The sensors that are used for fall detection can be classified into two categories (a) ambient sensors (b) wearable. The classification is based on the position of the sensor with respect to the user.

2.1.1. Ambient Sensors

Image, Audio and Vibration sensors are the major sensors that fall under this category. Fall detection using vibration sensors [9] is detailed. [10] describes a fall detection system that uses sound sensors. Vibration sensors are usually piezo-electric based and pick up vibrations caused by the fall of a person. The issue with using ambient sensors to detect vibration or audio samples is the large amount of noise present that requires pre-processing of signals using appropriate filters. If the fall detection system is part of the IoT system, then the complexity of the end device will be extremely high. Also, there is a possibility that soft falls may go undetected. Image Sensors [11] analyse images captured every few seconds to find if any person has fallen, in most cases multiple camera frames will be required. Multiple cameras can be placed in a single room and falls can be detected. Using multiple cameras in every room where the elderly live, is not feasible; especially considering that most elderly live on a restricted income.

All these sensors; vibration, acoustic and image are placed in and around the area where the elderly live and hence, they are termed ambient sensors.

2.1.2. Wearable Sensors

Wearable sensors are primarily IMU sensors such as accelerometers, gyroscopes and magnetometers. In some research work [12], GPS has also been used, while in the case of some, biometric parameters such as heart rates, $Sp{O}_2$ and skin temperature, etc were used.

In many cases, smartphones that already have an IMU sensor or smartwatches equipped with IMU sensors are used. GPS or biometric-based sensors in these devices are generally used for validations.

IMU sensors are made up of three different sensors (a) Accelerometers (b) Gyroscopes (c) Magnetometers.

The accelerometers will experience sudden changes in value during a fall. Based on the position of the sensors, in some cases wrist-worn, thigh-worn or torso worn; The amount of acceleration along the 3 axes will vary. The gyroscope measures the angular velocity along with 3 axes. The integral gives the angle of the person with respect to the ground. The third part of the IMU sensor, that is the magnetometer provides the orientation with respect to the Earth’s magnetic field. In case of falls, there will be a significant change in all these parameters. In some cases, heart rate sensors and skin temperature sensors are used as they may supplement the IMU data. Heart rate may increase in case of a fall, and so may the skin temperature. Wearable sensors are preferred over ambient sensors since they can move with the elderly also now IMU sensors are an integral part of smartphones and smartwatches which makes them an ideal choice; they are already available and will cost lesser even if a new system was to be built around them.

2.2. Data-Sets

Generally, to train and test the ML/DL algorithms large amount of data is required. There are multiple datasets available online, but they vary in terms of the type of sensors, manufacturer of the sensors, demographics of the train and test volunteers, the types of ADLs and Falls monitor, Sampling rate, duration of the data collected, number of volunteers and hence the number of data samples. The format of the data may be in terms of simple digital data from sensors, acoustic signals, or images. In the case of some datasets, raw data is available and in the case of some datasets, the features extracted are available. The feature extraction technique may also differ. Different datasets give different accuracies with different ML algorithms. Very little information is provided on why there are variations in accuracies. Does variation in gender, age, height, weight, and previous health issues affect accuracy is something that has not been analysed. Even when we run the ML/DL algorithms it is very difficult to analyse the accuracies as very little information is provided about the volunteers due to privacy issues. A table analysing various public datasets collected over the last few years is provided in Table 1.

3. Data Collection Methodology

There are multiple datasets available as described in the background section. There are multiple issues with the datasets:

• Very few public datasets with readings from multiple sensors are available. Most public datasets only have linear acceleration data.
• Very few datasets available that have a wide diversity in terms of age, gender, height, weight and health issues
• Even in datasets where there is diversity, no information is available on the ratio of gender, age, height or weight
• The number of volunteers is usually less. In most cases, less than 20.
• The list of ADLs and falls is not completely provided
• The details of how long each activity lasted are not available.
• The data collection methodology is not described
• The details of the sensors used are not provided; hence using multiple datasets becomes a major issue as they cannot be fused together.

3.1. Requirement for a new Dataset

Most datasets, such as Mobi Fall, UCI Fall, or SiS Fall, are either from the United States or European population-based. Also, datasets such as SmartWatch, SmartFall and Notch which we have used in our previous works, cover similar demographics. In other research works, the datasets are generally from China, which has a vast ageing population. At the time of publication of this paper, only one dataset is available from India [56]. This dataset is limited in terms of volunteers and activities performed, and the volunteers’ age range is restricted from 20 to 22 years. The models are heavily overfitted and gave an accuracy of 97% on the basic ML algorithms using just three statistical parameters, i.e. mean, median and Standard deviation. We needed a dataset that reflected the Indian population because several characteristics of Indians are entirely different from the European and World population. If we analyze some of the characteristics like

• Height – While the average world height of men is 5ft 9in, and women are 5ft 4 in, in India, the average height is 5ft 5in for the male population and just 5ft for the female population. In the case of certain European countries where the conditions of living and access to proper nutrition are good, the average height is even higher.
• Weight – While the World average of men is 89 kg and women is 77 kg, in India, the average weight of men is 65 kg and women are 55 kg.
• Lifestyles – While the European or the US population regularly exercise, regular exercise and diet ensure that the muscular and skeletal frame remains unaffected due to gradual ageing. However, In India, it is only in the last few years that exercise and diet have become a current trend. While we may have a healthy elderly population 40-45 years later, the ageing population is prone to significant changes in their skeletal and muscular frames. Hunchbacks are common among people even in their 50s in India [57]. The current elderly population in India is heavily dependent upon their male children for support. In the absence of financial aid, the elderly continue to work with existing health issues or are dependent on Government-run facilities which are overpopulated and understaffed. This necessitates the use of technology to monitor their health. Also, the level of literacy in the current ageing population of India is poor.

Our research involves building a low-cost device described in the introduction section, capable of detecting Falls and sounding alarms with or without network connectivity. The device not only collects data but also analyzes the data using ML algorithms to make required predictions. To train our device, we collected data from a wide range of Indian males and females.

In this work, the data was collected using our custom hardware worn on the wrist. As described in the Background of Work section, several works use multiple nodes placed across the body. Some of these devices are placed on the torso, some on the thigh and some on the ankle. We do not recommend placing the device on the torso, especially considering the frailty of the elderly in India, also several of them have a hunchback and might find the device extremely restricting. Placing it on the thigh would mean frequent removal or realignment whenever they wish to use the washroom, being a common place where a significant number of falls occur. This leaves us with ankle-based systems, which give very poor accuracies when the chest or waist sensors do not support them.

A low-power device that would require charging once a week and can be worn on the wrist would be highly convenient for the elderly. Hospitals and old age homes also use wrist-based tags. The device has been designed to be extremely lightweight and easily adjustable to varying wrist dimensions.

Though attempts have been made to compare datasets, not more than three or four [2] datasets are compared, and only certain statistical parameters are analysed. Besides, all these comparisons are made based only on accelerometer data. Due to these existing issues, we have collected our own data; this section gives the details of the volunteers diversity, the sensors and the data collection methodology. We also provide the details of the features extracted that are to be used by the various ML algorithms. The datasets are available at "https://shamanx86.github.io/fall_detection_data/".

Table 2 shows the comparison of various datasets that use accelerometers placed across the body.All the data are time-series data. It can be seen from the Table 2 that the BITS-2 dataset has the second maximum number of instances and covers a wide range of ADLs and Falls. This dataset has the maximum number of instances using custom wrist-based sensors only. As described earlier, our device is wrist-worn and all our sensors have been calibrated and their accuracies have been verified for wrist-based output.

3.2. Volunteers statistics

• No of volunteers: 41
• Age range: 18-50
• Number of female volunteers: 14.
• Weight: 50 Kg – 120 Kg.
• Height: 4ft 11 inches – 6ft 4 inches.
• Existing Health issues: High blood pressure, Diabetes, Hypertension Claustrophobia, there were some volunteers who were prone to panic attacks, sinusitis, sinus tachycardia, thyroid, malnutrition, hypochondria, extreme anxiety, low blood pressure, prostate, and early sign of arthritis.

The device that was used to collect the data was a prototype that was developed by us using a Qualcomm Snapdragon 820c SoC interfaced with medical-grade IMU(inertial measurement unit) sensors and a Heart- rate sensor. Figure 2. shows the diagram of the prototype of the device.

Table 3 gives the ADLs activities and Table 4 the Falls activities performed by the volunteers. The duration was precisely calculated using the inbuilt timer of the SoC, and the sensor values were captured using the same timer. This ensured that an exact sensor window of 2 minutes or 30 seconds, depending on the activity, was exactly followed. The activities that violated the time windows were cleaned and re-performed. The activities and the Falls have been selected based on the kind of impact the ADLs and the Falls will have on the sensors. Also, some of the ADLs, such as climbing up and climbing down, may result in a Fall. The Falls were sampled from a literature survey, and the most common falls that are experienced by the elderly were chosen for our research [58].

For convenience, the complete details of each volunteer are provided in Table 5. To maintain the privacy of the volunteers, user-ids are assigned.The volunteer statistics have been summarised in Table 6

The data was collected by asking the volunteers to perform the falls within an anechoic chamber over a period of three months; each volunteer spent roughly six hours performing the various activities. In the case of female volunteers, all activities and performances were supervised by a female faculty. As the anechoic chamber is padded with a thick sponge, during the falls, the volunteers landed on the soft material, and hence they were not injured. As a result, all falls ended up being soft falls. While the falls were performed inside the anechoic chamber, ADLs were performed under real-life situations, so climbing up for 2 minutes means a volunteer kept on climbing for 2 minutes. For this, we used a multi-storeyed building. Before every activity was performed, it was verified whether the volunteer’s heart rate had returned to their baseline values.

This was necessary since some of the activities, like climbing up and down the stairs, Jogging and Jumping, ended up increasing the heart rates of the volunteers. All volunteers had signed a no-objection certificate with two witnesses before performing the experiments. This was necessary to satisfy the ethics committee of BITS Pilani.

We coded the device to collect data from the sensors and transfer it directly into a csv file. A csv file was created for every activity for each user. Hence, we had a total of 24 csv files per user for 41 users in total.

In the next section, we describe how the data was cleaned, processed and extracted from the csv files.

4. Experimental methodology

4.1. Pre-Procesing of Raw Data

Data from each user for each activity was recorded on different comma-separated value(.csv) files. There were over 6480 CSV files. Each file was examined for blank entries and abnormal values, and those files were eliminated. Basic pre-processing also included removing data using data imputation using the last observation carried forward (LOCF). So, though we had more than 41 volunteers, only data from 41 volunteers were considered after cleaning.

Improving performance metrics of the ML models not only requires data pre-processing but also feature engineering and selection. Several of the researchers used windowing and feature extraction [59] [60] [61]. We have observed from the existing literature that ML classifiers are usually applied to single feature comparison [62] [63] [64].

The data set was partitioned into two sets, one containing IMU Features only and the other consisting of IMU and Heart-rate features. Each was split into training and testing datasets using the standard 70-30 split. We ran the algorithms with Raw data on Qualcomm Snapdragon 820c SoC. The SoC could not even converge for simple instance-based ML algorithms such as KNN; hence, we had to go in for feature extraction.

4.2. Feature Extraction

Feature engineering includes features scaling, feature extraction and selection. We did not apply feature scaling as the data was within the minimum-maximum range expected from the sensor. Feature extraction can be accomplished manually or by using automated methods. Training ML/DL directly with Raw data results in high data rates and information redundancies, leading to low accuracies.

Several types of features can be extracted from time-series data, and the most common among them is time-domain features, in contrast to frequency-domain features. Our data is time series data since its value changes over time. There are multiple statistical parameters. Since we were running the feature extraction on an 820c SoC and planned to immediately use the feature to make a prediction on the ML algorithm while running the same on the SoC. We used only the basic parameters such as mean, median, maximum, minimum, variance, and standard deviation. We also added kurtosis and Skew to the set of features. The RMS values of each parameter were calculated then we applied statistical feature extraction. In order to amplify the extracted information, the information was broken up into different segments, and then feature extraction was performed. A window size of 20 which mapped to a window interval of 5 seconds, gave us the best results. All points in the window had equal weightage. Subsequently, feature extraction was performed for every window.

In this paper, we have not used feature selection. However, some statistical parameters, such as quarantine, histogram and EDFC, did not give us productive results and were time-consuming on the SoC. Hence, they were not used. Our later experiments, which are not included in this paper, with feature selection, showed that only three parameters, mean, minimum and standard deviation were the statistical parameters that gave the most productive results with minimum latency. Data pruning is not included in this work, as we used feature selection and data pruning techniques only with ensemble algorithms. Here we are using the basic ML algorithms to understand how user characteristics affect the accuracy of the results.

In any ML-based system, whether health care or any other application such as NLP, Image analysis, etc. The train and the test data would be sourced from a completely different set of users. In the case of Geriatric Fall detection, the models are trained on a different set of users and used on a different set of users. While training, we used volunteers who are in a lesser age range, have lesser health issues and probably have a better lifestyle compared to the end-user of the fall detection device we built. In this paper, we tried to analyze the effect of specific user characteristics such as height, weight, gender, and pre-existing health conditions and their effect on the performance of the ML model. For instance, most volunteers (66%) were male, whereas the end user may be predominantly female, having been trained on a predominantly male population; even if an algorithm gives an accuracy of 97% in detecting falls, will this be maintained when the users are predominantly female? Indian ageing statistics [58] show that more than 70% of the elderly living alone in either poor conditions or in government-run elderly homes are single women who have lost their spouse and do not have any other family support. These women are the final target of the device we will use. Hence the device is low-cost and easy to use. The ease of use is essential as many of the elderly population in India currently have a low literacy level. It is not only gender, but other parameters also affect the accuracy and behaviour of the algorithms, especially height and weight, when considering IMU sensors. Referring to the table-6 in the previous section, the average heart rate is expected to be between 60-100 beats per minute; if we observe the baseline heart rate and reconfirm using medical grade sensors, there were users whose baseline heart rate was more significant than 100 bpm(8 out of 41), this considering the healthy population. The heart rate would be even more erratic with the elderly. How the heart rate affects the ML model’s performance is another question we are trying to answer in this paper. The paper’s aim is not to analyze the ML model but rather to analyze the effect of the user parameters on the productivity of the ML model. Will the device we have developed continue to perform when used by the target audience, or will its accuracy be very low as we train using completely different user demographics? This is the main reason why the experiment was conducted.

4.3. ML Algorithms

In order to analyze the effect of user characteristics on the behaviour of ML algorithms, we used five common ML algorithms. Our aim is not to evaluate the ML algorithms but rather to study the effect of user characteristics on the performance of the ML models. After going through multiple literatures, we went in for the most commonly used supervised ML algorithms (a) Naïve Bayes [65] (b) KNN [66] (c) Logistic regression [67] (d) Random Forest [68] (e)SVM(Suppport vector machine) [69].

Even though the Random forest is an ensemble algorithm, its complexity is comparatively lesser than AdaBoost or XGBoost. In AdaBoost, latencies are very high and hence difficult to converge in 820c, whereas XGBoost requires good computational power. Each of our users had 112 features for each activity which would have made the implementation of XGBoost extremely complex. Supervised learning can be used in a classification problem or a regression problem. Mostly, in healthcare, it tends to be a classification problem, especially in the case of our research, activity is to be classified as a fall or a non-fall. Commonly used supervised algorithms are

4.3.1. Decision Tree(Random Forest)

is a technique for approximating discrete value target function, representing the learnt function as a decision tree [70]. A decision tree classifies instances by sorting them from root to some leaf nodes based on features. While using a decision tree, the focus is on deciding which feature is the best classifier at each node level. Classification of instances starts at the root node, called the decision node. Based on the value of the decision node, the tree traverses down along the edge, corresponding to the value of the output that a feature will generate. This continues until the leaf node is reached. Several algorithms are available to implement decision trees. The most common are classification and regression trees, iterative dichotomised-3 (ID3) [71], Automatic interaction detection, and CHAID.

The decision trees fall under an Ensemble method termed Bagging; Random forest is an extension of Bagging as it utilizes bagging and feature randomness to create an uncorrelated forest of decision trees. Random forest algorithms have three primary hyperparameters (i) maximum depth, (ii)the number of trees, and (iii) the number of features.

We tried for different values of maximum depth. If the number of splits is too small, the model under fits the data, and if it becomes too high, the model overfits. . Usually, the values 3, 5 and 7 are preferred, and the fourth option is “None”. If “None” is used, nodes are expanded till all leaves are pure or until all leaves contain less than minimum_samples_split, which is also one of the parameters programable in random forest. We got the best result when this parameter was set to 2. The best result was obtained when the maximum depth was set to “None”. We also varied the number of trees, starting from 50 to 200. The best result was obtained at 170 trees. We also varied the max_features parameter. It has three possible values “sqrt”, “log2”, and “None”. When “None” is used, the number of max_features will be equal to the number of features which in our case is 121. We obtained the best result for sqrt, and hence parameter was set to “sqrt”.

4.3.2. Naïve Bayes

Naïve Bayes (NB)uses Bayes theorem of probability. Bayes theorem calculates the posterior probability of an event X, given some prior probability of event Y, represented by

$$\mathrm{P}(\mathrm{X}/\mathrm{Y})=\left[P\right(Y/X)*P(X\left)\right]/P\left(Y\right)$$

(1)

Naïve Bayes classifier falls under the category of simple probabilistic classifiers. Bayes theorem assumes independence among the features, which is not the case for the data we use to classify falls [72][73].

4.3.3. Support Vector Nachine

Support Vector Machine(SVM) is used for classification and regression. It works on the principal calculating a margin. In this algorithm, every data item is plotted as a point in n – dimension space when n is the number of features in our dataset. It classifies the data into different classes by finding a line(hyperplane) which separates the training datasets into classes. It maximises the distance between the nearest data point (in both classes) and the hyperplane [74].

In SVM, the programmable parameters are the Kernel, Gamma and Regularization parameter “C”. The Kernel can be either “linear”, “poly”,” RBF”, or “sigmoid”.The best results were obtained for “RBF”. In the case of “linear” and “poly”, the drop in accuracy was about 10%, and in the case of “sigmoid”, the drop in accuracy was about 30%. The Kernel parameter helps determine the shape of the hyperplane and the decision boundary. For ”rbf”, the radial basis function gave us the best result.

Gamma can be either “Scale”, “Auto”, or “float”. We tried for various values of “float”; we also tried “auto”, but the best results were obtained for “Scale”. The Gamma parameter determines how far the influence of a single training example reaches. If Gamma is small, the model will be too constrained. The radius of the “rbf” kernel alone acts as an excellent structural regulariser. Using high values of “C” will not help further. We varied the regularisation parameter “C” from C=1 to 15; the best results were obtained for C=10, after which the accuracy plateaued till C=15, after which the accuracy started dropping again. The “C” parameter tells SVM optimisation how much misclassifying must be avoided in each training example. For large values of C, the optimisation will choose a smaller margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. A small value of “C” might misclassify in a training dataset.

4.3.4. Regression Analysis

It is a predicting modelling technique that establishes a relation between a target and an independent variable, i.e. the predictor. In this method, we fit the line/curve through the data points to minimize the differences between the distances of the data point. Various kinds of regression analysis exist, such as linear, logistical and polynomial. In the case of Fall detection, logistical regression is used [75]

4.3.5. K-Nearest Neighbour

It’s an example of instance-based learning. It is a non-parametric method used for both classification and regression. Given n-training vectors, the KNN algorithm [76] identifies the k nearest neighbours of an unknown feature vector whose class is to be identified. The ideal “K” value for KNN was found to be nine while using “Minkowski” as a distance metric.

4.4. Analysis of Effect of User parameters on the Accuracy

To understand the variations in accuracies we used various combinations of the user demographics. This was done to understand the effect of user parameters such as gender, age, height, weight and health issues on the accuracies. We also analysed which of the ADLs and Falls were incorrectly detected with respect to the user parameters. We used a combination of user demographics to understand the behaviour of the algorithm and why there were variations in Accuracy, Sensitivity, Specificity and False Negative rates. We used a combination of varying user parameters for training and testing. For example, to understand the effect of gender on the test accuracies, we first ran the ML algorithms separately for male volunteers using their data for both training and testing. We repeated the same for the female volunteers. Then we used the data of the female volunteers for training and the male volunteers for testing and vice-versa. Many of the existing public datasets which are used for training the ML algorithms have similar user demographics. In most cases the volunteers are male and in the age between 20 and 30 years with no known health issues. The actual users of the end product will be people above 60 years, both male and female with several pre-existing health conditions. Under these circumstances, whether the fall prediction algorithms will work accurately needs to be analysed. Hence in this paper, we have tried to analyse the behaviour of the algorithms with varying user demographics. The combinations that we used for testing and training are given in Table 7.

We ran the algorithms with and without the data from heart rate sensors to understand the effects of biometric parameters during falls. In the next section, we give the complete results and the analytics.

5. Results and Discussion

5.1. Overall Performance Analysis for various ML algorithms with varying data sizes

We initially analysed the effect of varying data sizes on the performance of ML algorithms before analysing the effect of user demographics on the performance of the algorithms. As mentioned in earlier sections we have used the common ML models (a) Naïve Bayes, (b) KNN, (c) Logistic Regression, (d) SVM and (e) Random Forest.

The accuracy, specificity and sensitivity of various algorithms with (a) 41 users (b) 35 users (c) 30 users (d) 25 users (e) 20 users and (f) 10 users are shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8.

The best results among all the algorithms were obtained when heart rate was included, as can be seen in Figure 3 and Figure 5. The best result was obtained for 30 users for all algorithms, except for random forest (where the best result was obtained at 10 users), with heart rate the peak accuracy was 98.5% obtained by SVM and 96.5% without heart rate again in SVM. For all user combinations, with and without heart rate the sensitivities were higher than the specificity. This is as per the requirement for Fall detection. Since we are dealing with geriatrics a false positive is better than a false negative.

The main activities that were incorrectly detected were those of User-1 as the subject had an erratic heart rate due to a medical condition known as Sinus Tachycardia. In the case of other users, very few ADL or Fall activities failed since we had selected the 10, 20, 25, 30 and 35 users randomly out of the total 41 users. Wherever user-1 was not selected, we had better results. When heart rate was ignored several of the ADL activities which were related to “lying on the bed” or “getting up slowly/quickly from the bed” were incorrectly detected as falls depending purely on the IMU sensor’s values. Also “backward falls and falls from the bed” were incorrectly detected as ADL activities for several users. When heart rate was ignored more of User – 1 activities were incorrectly detected especially since several of the ADLs were construed as false positives. This included “lying on the bed”, “lying on the back and getting up slowly/quickly” and “transitioning sideways to one’s back while lying down”. This caused a drop in accuracy from 99% to 98%, especially due to a drop in specificity values from 0.99 to 0.98. It can be concluded from the results that when we use the heart rate of a volunteer with severe health conditions the accuracies obtained are better especially since there will be a huge spike in heart rate while performing Fall activities as against performing ADL activities. The sensitivity was higher by almost 1.5% when heart rate was used and the specificity was high when heart rate was ignored.

Table 8 and Table 9 show the accuracy, sensitivity and specificity for the various age groups “under 30”, “30-40” and “40-50” for individual algorithms with and without heart rate. As usual, Naïve Bayes gives the lowest accuracy and SVM gives the highest accuracy. When training and testing were done with the same age group, it can be observed from the table that the impact of accuracy is not much. The accuracies have dropped in the case of all the algorithms as the size of the dataset would have been reduced as we only considered volunteers under the age of 30. While the accuracies of SVM with the whole dataset went up to 98.5%, here the highest accuracy again being produced by SVM is 95.10% for the age group of >30, in the case of 30-40 it drops to 87.50% as the number of volunteers in this age group was lesser. At 40-50 it was higher at 90.91%. In the case of 40-50, Random Forest and Logistic Regression gave the highest accuracies at 93.18%. Random Forest’s Decision trees work equally well with large as well as smaller data sets. When the simulations were run with volunteers under 30, heart rate had a huge impact for the higher age range which is 40-50 with Random Forest giving an accuracy of 99.31%. But in the case of 30-40, the IMU sensors had a higher impact. Since in most scenarios in practical life, volunteers would be in the age group of below 30 and users will be in the range of 50 and above; The impact of heart-rate is very significant. When we analysed which of the users were causing a fall in accuracy, they were primarily users in the age range of 30-40 but though they were physically fit, they had pre-existing health conditions and were on medications. Primarily user 30 with a pre-existing prostate condition and user 35 with a pre-existing diabetic condition.

Table 10 and Table 11 show the accuracy, sensitivity and specificity of the male and female participants for individual ML algorithms with and without heart rate. From the table, it can be seen that we get better accuracies with heart rate in the case of males but in the case of females, the accuracies fall with heart rate. In the case of females, the accuracies seem to increase or drop erratically because the number of females was only a little more than 1/3rd of the entire set of volunteers. When trained with the Male data set and tested with the Female, we got better results both in the case of With and Without heart rate, because the training was done with a larger population than the number of test subjects. This shows that the size of the data set has a huge impact while gender does not have a large impact on accuracy. So it’s very important to train the models with large datasets with the gender being insignificant. The train set must be larger than the test set.

Table 12 and Table 13 show the accuracy, sensitivity and specificity for subjects with pre-existing health conditions and subjects who did not have any health conditions were used as participants. Table 12 shows the result with heart rate whereas Table 13 shows the result without heart rate.

Of the 41 volunteers, 12 of them had pre-existing health conditions that varied from high blood pressure, diabetes, claustrophobia, vertigo, sinus tachycardia, prostate and obesity, etc. To study the effect of health conditions on the prediction we ran the algorithm separately for the volunteers with health issues and without health issues. The performance of the algorithms was better when heart rate was associated with people who did not have any pre-existing health issues. When we tried training the algorithm using people with health issues and testing them with volunteers who did not have any health issues and vice-versa, the heart rate had no such impact in the first case however in the second case, there was a slight improvement in accuracy as can be seen in Table 12 and Table 13.

Based on this we can come to a conclusion that the presence of health issues does affect the accuracy of fall detection and since erratic heart rate is usually associated with health issues, there is more possibility of errors when heart rate is used as one of the features to detect falls. This will prove to be a major hurdle as we move forward in our research since there are very few elderly who do not have any pre-existing health conditions, in fact in our 40-50 age group only one volunteer had no health issue or was not under any form of medication.

Table 14 and Table 15 show the accuracy, sensitivity and specificity for the subjects having height below 5ft 5in and subjects having height above 5ft 5in for data with and without heart rate respectively. The behaviour of the model would definitely be impacted by the height of the person as the amount of change in IMU values would be dependent on the person’s height. For a person whose height is above 5ft 5in, which is also the average height of a person of Indian origin, there is no effect of heart rate on the accuracy and the accuracy is completely dependent on the values derived from the IMU sensors. For people who are lesser than 5ft 5in the accuracy drops when heart rate is used. Even otherwise accuracies for people of height more than 5ft 5in are slightly higher than the volunteers whose height is lesser than 5ft 5in. With progression in age, the skeletal structure of a person may change introducing conditions such as hunchbacks which may cause a reduction in height, and hence the accuracies will be affected. When we trained using volunteers of height greater than 5ft5in and tested with volunteers with height less than 5ft 5in, there was a definite drop in accuracy both with and without heart rate. Whereas when we trained with people of height less than 5ft5in and tested using volunteers of height greater than 5ft5in, in the case of some of the ML models, there was a slight increase in accuracy with and without heart rate. This shows that the IMU sensors produce drastically different values that vary with the height of the person. Though we are extracting statistical parameters, the height of the person has a huge impact on the accuracy of the model. Hence, if we use volunteers who are young and in good physical condition and have an above average height to train the ML models and use it on elderly who will have pre-existing health issues, weakening of the muscular structure and reduced height, several of the ADLs may be misinterpreted as falls because the drop in accuracy here is due to the drop in specificity rather than sensitivity. This might be acceptable because in the case of the elderly, it is better to err on the side of caution.

Table 16 and Table 17 show the accuracy, sensitivity and specificity for the subjects having weight in the range of 50-65 kgs, subjects having weight in the range of 65 to 80, subjects having weight in the range of 80 to 120 Kgs with and without heart rates respectively. The impact of the weight of the subject on accuracies is very high, with accuracies for lower weight range i.e 50-65 Kgs being high and accuracy drops with an increase in weight. Also, the impact of heart rate on accuracy is visible only in the weight ranges of 80-120 kgs. As the weight of the person will have an impact on the heart rate as he/she performs stressful activities. Again cross-testing and training have the least impact on the weight ranges of 50-65 Kgs. When trained on the weight range of 50-65 Kgs and tested against ranges 65-80 and 80-120, the accuracies are not severely affected. But training with other weight ranges affects the accuracy as can be seen in the table. The drop in the accuracies in the weight range 65-80 is mainly due to the drop in specificity rather than sensitivity. When the weight is in the range of 50-65 which is usually below the Indian average or 80-120 which is above the Indian average, most of the ADL activities are detected correctly except in the case of Naïve Bayes where the sensitivity is better than specificity, but NB is more of a threshold based algorithm and its accuracies are generally very low. Again the elderly are prone to be frail with their weight being less than the average. Hence, we can get better accuracies irrespective of the age group used for training and testing.

6. Conclusion

Though there a multiple data sets available, and we have used them in our previous works, we were unable to interpret the reasons for the rise or fall in accuracy. Hence we collected our own data and analyzed it. In this paper, we have attempted to analyze the effect of various characteristics such as age, height, weight, gender and health issues on the accuracy of the various well-known ML algorithms. In geriatric fall detection, it is given that the test volunteers and the actual users will belong to different categories. While the test subjects are usually in the age range of 20-35 in most data sets, we stretched the age range between 20-50, but the actual users would be the elderly or the super elderly. While we did have some users who had serious existing health conditions and were on medication and we did find a fall in accuracy when we trained using healthy volunteers and tested the accuracy for volunteers with health issues. The loss in accuracy was primarily due to a drop in specificity hence some of the ADLs were being incorrectly interpreted. In some cases, the use of heart rate helped improve the accuracy in some cases the impact of heart rate caused a drop in accuracy. But we still recommend the use of heart rate as one of the parameters as any changes in heart rate in the elderly supported by the data from IMU sensors may be indicative of a fall.

The problem with multiple datasets is also that the demographics of the users tend to change with countries. In countries with a good health infrastructure and where healthy living is promoted, the ageing process is more graceful. The average height and weight also vary across countries. In India, healthy living and exercising are new concepts, overindulgence is a norm here; hence most elderly India suffer from various health conditions, and very few of them have access to good healthcare services due to the cost. Falls may times go undetected; repeated falls in the elderly are also very common in India. We need data sets that represent the Indian demographics. We have attempted to create such a database, and we have analyzed the database in detail for various user parameters such as age, gender, health issues, height and weight.

The number of features we have is 100,208 after feature extraction, we also plan to add certain parameters such as height and weight that have a huge impact on the accuracy as part of the features. We were also able to identify certain users who were outliers, this could be used to clean the data set further, we have currently retained the outliers as there may be final users who may have similar characteristics.

We plan to run ensemble ML algorithms on the end device built around Qualcomm Snapdragon 820c. This means features have to be pruned and the ensemble ML algorithms compressed so that the latency in obtaining the prediction will be much lesser than the sampling rate, and timely alerts can be issued.

Understanding the impact of features and user characteristics is the first step to pruning features which is what we have attempted in this paper. We collected data that is specific to the local user demographics, extracted relevant features and analysed the performance of the ML algorithms on the collected data.

A perfect data set would be one where the demographics are uniform and similar to the end user demographics. While some public data sets have attempted to use the elderly to do a limited number of ADLs, they have not been able to show a significant improvement in accuracy. Hence, we have to proceed with skewed data sets. The application of ranking and pruning of features gains importance. This is the next step in our work. This is required as we plan to use Dew computing IoT model.