Empowering Person with Disabilities in Smart Home Using Predictive Informing

1. Introduction

Today's technological development in the field of assistive technology and the environment of person with disabilities brings significant changes in the way person with disabilities live and participate in everyday life [1]. Assistive technologies are not representation of only information and communication devices and systems, but also services aimed at improving the functional abilities of person with different types of disabilities [2]. Having a disability can impact a person's ability to conduct daily tasks and fully participate in various environments, including their own home. However, everyday activities within their home such as cooking, cleaning, safe movement around the home or health monitoring can be integrated into a smart home environment where users can control individual devices [3].

A smart home can be defined as the interaction of information and communication technologies and services with the user through the home network to create better living conditions and environments for the user [4]. Today's development of smart homes and indoor environments intended for person with disabilities is based on the concepts of Ambient Assisted Living (AAL) and Enhanced Living Environments (ELE). The technologies and working principles used in the mentioned concepts are based on modern information and communication technologies such as Cloud Computing (CC) (Fog, Edge, Dew), Internet of Things (IoT), Machine to Machine (M2M), and diverse types of sensor networks and corresponding sensors [5]. Several different communication devices and systems are currently in daily use, and in most cases, the devices are used for multiple purposes and solve more than just one need [6]. The diversity of technologies available to support the autonomy and independence of person with disabilities offers the possibility of choosing a particular technological solution or set of solutions that are most suitable for their functional abilities [7]. According to the data on the quantity of IoT devices used in smart environments until the year 2025, there are 71 billion installed IoT devices expected, whereas in smartphone environments 12.86 billion devices are expected [8]. Following the development trends in the field of AAL and ELE concepts, the need for the application of diverse types of sensors, robotics, IoT, and voice assistants to manage them is increasingly emerging [9,10]. The context of the use of AAL and ELE concepts is presented through a literature review, where the authors stated the possibility of use and purpose in closed spaces and the importance of network heterogeneity [11]. The AAL concept, whose goal is to help and provide real-time information to elderly person and person with disabilities, is also increasingly applied in the design of new services based on decision support systems [12]. To model IoT services and deliver real-time information to person with disabilities, it is possible to use a conceptual model of system architecture applicable to an IoT environment [13]. According to a research paper [14], incorporating IoT technology in a smart home can greatly benefit visually impaired individuals by facilitating their daily activities. This, in turn, enhances the overall autonomy, safety and comfort of users in their own home environment.

The development of technology such as artificial intelligence (AI) provides the ability to detect and monitor the daily activities of person with disabilities (users) in their homes. In the user's smart home environment, the following system components are important: appropriate hardware equipment for data collection (sensors and actuators), a communication network (sensor autonomous network), a set of applications and tools for data processing and management, and data storage equipment [3]. The e-care@home system defines certain data sets and the modularity of the smart home system, which enables use regardless of the type of impairment the user has [15]. Also, the application of AI technology and 6G network is applied in medicine as a closed user space where appropriate IoT services (IoMT) are applied [16]. The application of autonomous and intelligent networks and systems is today increasingly used also indoors, mostly in the field of eHealth and in providing services for the senior persons and persons with disabilities [17,18].

In addition to the application of AI technology and the environment of a smart home for person with disabilities, it is also possible to use it for elderly users who also have certain health problems, such as dementia and an increased risk of falling when moving. For this purpose, the authors propose the improvement of data sets with the aim of a more efficient system for detecting user falls and informing caregivers [19]. AI requires the definition and use of certain types of data sets and thus the creation of user forms for which it is also necessary to define user needs. Daily activities depend on the distinctive characteristics or types of user's impairment and frequently require an individual approach to solving daily needs. For this purpose, to model innovative services in a sustainable ecosystem, it is necessary to clearly define data sets, all to deliver accurate information to the end user and the possibility of predictive information.

User needs are the basis for defining data sets, but also for designing a smart home according to user requirements [6,20,21,22]. There are 624,019 persons with disabilities in the Republic of Croatia, of which 353,550 are male (56.7%), and 270,469 are female (43.3%), thus persons with disabilities make up about 16.0% of the total population. Groups of users and their number in the Republic of Croatia according to the type of impairment that is the subject of this research are locomotor system impairment - 177,547, speech-voice communication impairment - 51,149, visual impairment - 20,526 and hearing impairment - 17,679 [23].

Following on from the above, the author's motivation for this paper is to propose creative solutions and services to augment the living standards of those with disabilities in a smart home environment. The primary aim of this paper is to highlight the complex interdependence of the different elements of a smart home system and underscore the importance of defining data sets and creating user profiles depending on the type of impairment of the user.

2. Smart Home Concept for the Person with Disabilities

Research [24] explored the lifetime home standards (LHTS) of individuals with disabilities. The conclusions are mainly focused on the architectural adaptation of the environment in which the user is located. The research also highlighted the importance of incorporating technological solutions to enhance their living conditions. However, high-tech solutions such as electronic doors and windows remain financially inaccessible to most users. Nonetheless, users expressed that implementing such solutions would greatly improve their quality of life. Safety within the home was also emphasized, particularly the need to detect electric shocks and gas leaks as well as the importance of lighting management. Another research examined the benefits of home automation systems for visually impaired individuals [25]. The research results point out that there is a strong interest in home automation to increase the autonomy of visually impaired individuals, with lighting control and heating systems being the most prominent areas of focus.

To design the conceptual system architecture, the smart home environment for the visually impaired can be considered through three segments: user needs, home environment or architecture, as well as the technology used in the home [3]. This approach can also be applied to other groups of persons with disabilities, where it is important to show the characteristics of impairment as shown in Table 1.

Based on the type of user impairment, in Table 2 are defined increased risks depending on the smart home usage scenario (kitchen - K, living room - LR, bedroom - SR, bathroom/toilet - TWC, terrace/balcony - TB and daily activities - DA).

Data sets will be defined as elements of the smart home management system based on the above-mentioned increased risks for certain groups of users and usage scenarios. It is also important to emphasize the communications segment that is necessary for the operation of an automated system based on generative AI. The sensors and function tags used in the smart home environment are defined through three subsets, shown in Table 3.

Within each subset, defined sensors collect the data needed to deliver the information requested by the user. Sensors are defined as variables and have their labels and are shown with a Venn diagram and a mathematical model (Figure 1) [13]. The conceptual mathematical model for providing information (predictive information) to the user is presented as a union of defined sets, i.e., data collected from around the smart home.

$S_{shome}=S_{hi}\cup S_{ii}\cup S_{ei}$ or if it is an expansion with additional subsystems

$S_{shome}=\bigcup _{i=1}^S{S}_i$

from which:

• $S_{hi}=\left\{S_p,S_{pu},S_s,S_t,S_b,S_{bu},S_{fa},S_{el},S_d,S_{ao}\right\}$
• $S_{ii}=\left\{S_f,S_g,S_{fa},S_{fl},S_{el},{S_{er},S}_{bu}\right\}$
• $S_{ei}=\left\{S_{od},S_{ai},S_{fg},S_l,{S_{te},S_h,S_d,S_{ao},S_{ocd},S_f,S_{fl},S}_{el},S_{fa},S_{er}\right\}$
• Dependence of individual variables in several subsystems:
• $S_{hi}\bigcap S_{ii}\bigcap S_{ei}=\left\{S_{fa},S_{el}\right\}$
• $S_{hi}\bigcap S_{ii}=\left\{S_{bu}\right\}$
• $S_{ii}\bigcap S_{ei}=\left\{S_f,S_{fl},S_{er}\right\}$
• $S_{hi}\bigcap S_{ei}=\left\{S_{ao},S_d\right\}$

The graphic representation in Figure 1 shows the inclusion of individual data (variables) from all these subsystems. Fall detection (S_fa) and electric shock (S_el) are connected to all three subsystems because the delivery of information about the user's health (state in which he is and type of impairment) and location must be real-time according to the emergency services or the caregivers if this has been previously defined. Burn detection (S_bu) on the user is identified by sensors from the subset of user health (S_hi) and incident information (S_ii) to call emergency services. When a flood (S_fl), fire (S_f), and earthquake (S_er) are detected from the environment where the user is located, a call to the emergency services or the caregiver is also made. To preserve the user's health (allergic reactions), the sensor of dust (S_d) and air quality in the external environment (S_ao) informs the user about this. All defined variables represent a set of user data necessary for the creation of real-time information and the possibility of predictive user information.

The concept of a smart home for person with disabilities and associated sensors is shown in Figure 2. The figure shows all the sensors listed in Table 3. Inside the smart home, the user can be informed by voice about the time (hours and minutes), reminder management, and detection based on data on incidents, vibration (earthquake), flood, etc. The device can be used to control TV and radio devices, refrigerators, vacuum cleaners, air conditioners, front doors, blinds, cameras and other available IoT devices. It is also possible to provide information to the user about health, during which it is possible to monitor the user's pulse, pressure, temperature, amount of sugar and oxygen in the blood. For this purpose, the user should have a smart bracelet that is equipped with appropriate sensors. The smart bracelet, with its characteristics, can be modular, i.e., adaptable to the user's type of impairment. Depending on the type of impairment, appropriate sensors are activated and the format of providing information to the user is adjusted [26]. The connection of the bracelet with the smart home system is also required to create a user base.

The concept of a smart home for person with disabilities and associated sensors is shown in Figure 2. The figure shows all the sensors listed in Table 3. Inside the smart home, the user can be informed by voice about the time (hours and minutes), reminder management, and detection based on data on incidents, vibration (earthquake), flood, etc. The device can be used to control TV and radio devices, refrigerators, vacuum cleaners, air conditioners, front doors, blinds, cameras and other available IoT devices. It is also possible to provide information to the user about health, during which it is possible to monitor the user's pulse, pressure, temperature, amount of sugar and oxygen in the blood. For this purpose, the user should have a smart bracelet that is equipped with appropriate sensors. The smart bracelet, with its characteristics, can be modular, i.e., adaptable to the user's type of impairment. Depending on the type of impairment, appropriate sensors are activated and the format of providing information to the user is adjusted [26]. The connection of the bracelet with the smart home system is also required to create a user base.

Inside the smart home, there are sensors with a safer and more comfortable execution of everyday activities. The virtual assistant provides voice control of smart IoT devices and provides the user with information collected from sensors. Sensor information is delivered upon user request. For the mentioned concept, it is important to use an application solution made according to the guidelines of universal design. The idea of the presented concept is to enable the creation of a user base of the user's needs to provide accurate and real-time information and predictive user information.

3. Taxonomy of Communication Technologies and Devices

To apply AI technology, i.e., perception of the environment in which the user is located and provision of relevant information, a taxonomy of communication technologies and devices shown in Figure 3 is proposed. The purpose of using AI is to, based on collected data from sensors that monitor everyday situations, and movements, the user's behaviour and obligations are created by appropriate user patterns as a basis for predictive user information. User forms creation (depending on the type of impairment) defines the basis for modelling IoT services based on AI technology, all to develop a predictive model for information delivery.

For service delivery to the user, the smart home environment is divided into four layers of automation: Perception and Actuations, Network/Transmission, Preprocessing and Middleware, and Applications and Devices. The Perception and Actuations layer represents physical objects that serve to collect information from the user's environment. The user's environment is made up of defined scenarios in which he finds herself every day (kitchen, living room, bedroom, toilet, and outdoor balcony/terrace), as well as the user's health and daily needs.

The user's health condition is monitored via a smart bracelet equipped with body sensors. Body sensors are a group of sensors that collect data: heart rate, pressure, sugar, temperature, and breathing. Also, to monitor the user's health, a sensor is used to detect burns, electric shock, and sensors that can indicate possible allergic reactions (dust sensor and air quality in the external environment). Sensors that are used to warn about the possible occurrence of an accident and belong to the group for providing incident information have the role of detecting: fire, gas (CO₂), user fall, flood, electric shock, earthquake, and burns. To monitor user activity in all scenarios, sensors are used for objects/obstacles detection, monitoring of air quality in the indoor environment, food quality, detection of light intensity, temperature measurements in the home, and measurements of the amount of moisture and dust.

Network layer or Transmission layer is responsible for secure transmission of data obtained from the sensor devices to the processing system, i.e., to the Preprocessing and Middleware layer through various network technologies. It consists of an access sublayer that has the role of collecting data from the Perception layer and of sending it to the Internet sublayer. In the access sublayer, which is in the part of the core network, the key role is played by the technologies Z-Wave, Zigbee, Bluetooth BLE, GPS, and RFID. Wi-Fi technology is used to transfer the obtained information from the access sublayer to the next layer, Preprocessing and Middleware [3].

Preprocessing and Middleware layer is a software layer that can directly synchronize the services with the appropriate requirements and has a connection to the database. It performs processes such as preparing the information for further processing, data cleaning, standardization, values of data within a dataset, balancing, intelligent routing and translating network addresses. It is based on CC because of the great benefits it provides, such as scalability and flexibility. In the proposal of this paper, the user data are saved in the user database that is stored in the CC infrastructure. The preparation of information for delivery in the appropriate form is enabled using M2H (Machine–to–Human) and M2M technologies. The ML (Machine learning) algorithms enable intelligent adaptation of applications to changes in the environment, and the AI technology helps to collect data from the devices, predicts the user behaviour within the smart home environment and helps improve data security [3].

Based on the processed data in the Preprocessing and Middleware layer, the Application and Devices layer is responsible for providing end-user services and the proper operation of applications and devices. Identification, processing, and storage of data are performed in CC models depending on the level of service used. In addition to sensors and actuators (S_n+1 and A_n+1), devices (D_n+1) are also used, which by their characteristics belong to the group of IoT devices and have been integrated into the smart home system. Devices include smart robots, air conditioners, automatic doors at the home and rooms entrance, smart TV, refrigerator, cameras, virtual assistant, and smart home management systems. Application solutions on smartphones and smart home management systems are also used. API (Application Programming Interface) is used to provide the appropriate form of information that is defined through the user profile (degree of impairment, etc.).

4. Presentation of the Interaction of the Elements of the Smart Home System

According to the defined sensors and the communication network used in the smart home environment and the defined scenarios of the application of the smart home service, Figure 4 shows the interaction of the elements of the smart home system. The communication of the elements is based on the AAL/ELE concept, the main purpose of which is to work in closed spaces adapted to user needs [10,11,27].

The role of sensors and actuators (S_n+1 and A_n+1) is to collect data from the user's environment depending on the scenario of using the smart home service. Devices (D_n+1) used in the smart home environment also generate data sets in the IoT hub and forward them to the CC, whose role is to collect, identify, and process data. The interaction between all levels of communication towards the CC depends on the service-level agreement (SLA), which must satisfy the corresponding level of service. The data is stored in the database depending on the identification of the associated set (function dependencies – S_hi , S_ii and S_ei). An example of detecting an elevated pressure level in a user and performing tasks is shown in Figure 5.

When detecting pressure, it is important to set the pressure levels (normal - high). The mentioned levels also represent a weighting factor that is crucial for the execution of tasks and activities. One of the tasks called Emergency call consists of several activities that can be performed in different usage scenarios (K, LR, SR, TWC, TB, DA) depending on the type of user impairment (activities such as contacting emergency services in case of fire, flood, fall, injury). At the same time, the activities represent the way and the possibility of delivering information to the user or one of the defined stakeholders, and their sets are defined as tasks that are performed through specific devices.

Through the Application Programming Interface (API), information is displayed to the user/stakeholder in a specific format. Each piece of information is adapted to the user (depending on the type of impairment) using M2H communication in an acceptable form. To collect data on users, it is possible to use a separate database where the user data needed to define user profiles (type of impairment and method of data delivery) are located. The content is adjusted based on the user's characteristics and the functionalities selected by the individual user and is presented in Table 4.

For persons with locomotor impairments, it is important to consider all the above-mentioned guidelines if it is an additional type of impairment, however, for persons who move with the help of wheelchairs, the aspect of providing accurate information and communication with the caregiver or emergency services is important. When identifying an incident situation, a person must have the simplest viable way out of the home environment. person with impaired vision and physically impaired person who use wheelchairs and who do not have speech or hearing problems can use the services of a voice assistant in a smart home environment. Its role is the ability to manage IoT devices in the home and predictive user information.

5. Testing and Performance Results of Sensors in a Smart Home Environment

To verify the effectiveness of proposed mathematical model, which aims to define information for users that are in smart home environment, specific sensors and IoT equipment were utilized. The data was collected and processed in the Thingspeak cloud environment for data storage and continuous monitoring of information from the user's environment. Figure 6 shows the Thingspeak environment and data collection time in laboratory conditions. Data was collected at average intervals of 15 [s]. Each record includes a unique identifier (id), time interval between measurements, sensor value, latitude, longitude, elevation, and status (2023-08-29T16:14:05+02:00,15,28.00000,82.39999,40.00000).

The collected values are shown in Figure 6, for example, temperature value from 26°C to 28°C, humidity amount from 20% to 40%, fall detection (motion), and recognition of objects at a certain distance from 3 [m] to 15 [m]. The equipment used for simulation and testing is shown in Table 5. The test was conducted in the Laboratory of Development and Research of Information and Communication Assistive Technology at the Faculty of Transport and Traffic Sciences, University of Zagreb.

The sensors collected various values including fire, burn, CO2, earthquake, object, temperature, humidity, and fall. These values were then stored in the Cloud Thingspeak environment and later in the MYSQL database as shown in Figure 7. The database was created based on the mathematical model that described the type of data and values. The tables that collected the sensor data were linked to the table responsible for storing and delivering information to the end user.

To deliver information in a suitable format for a specific user, such as a blind or partially sighted person with unique characteristics, an appropriate API must be defined in the middleware layer. In this case, an API can be used to receive data and send configuration values to an API that delivers information in the appropriate format for the user's device, regardless of the device type.

Testing of sensor performance and storage of collected data according to defined sets S_hi, S_ii and S_ei showed the importance of storing individual values in precisely defined sets. These sets are essential in defining user profiles, which are necessary for the development of ML models. For instance, in a 15-day period, 80,398 data points were collected. Using a conceptual mathematical model and defined variables, the value of generating information about the fire is shown:

• S_{info_fire} = {S_ii, S_ei, S_hi}

Based on the information above, when reporting a fire, incident information (S_ii) is provided along with the ability to call emergency services. The user's environment (S_ei) also provides information about the user's location within different scenarios (K, LR, SR, TWC, TB, DA) and whether the user is in danger (S_hi). This method has been found to be highly successful in storing data based on distinct variables and sets, enabling efficient delivery to the user and possible interest groups (stakeholders)

6. Conclusions

The development of assistive technologies and the application of AI to help person with disabilities today can be said to be not just a type of device, but a set of methods, services, and an entire ecosystem that must be sustainable and accessible. Services based on IoT technology and AI can raise the user's level of quality of life, which increases the degree of mobility and involvement in all spheres of social life and everyday life. Designing and modelling such systems should be viewed according to user needs and based on that, create a sustainable and accessible ecosystem. The possibilities provided today by information and communication networks, and technologies such as AI, ML, and autonomous systems can create a faster and more efficient system for the user. Such possibilities provide predictive user information and interaction with the environment in which they are located.

Based on the proposed taxonomy from this paper, it is possible to create a smart home for person with disabilities depending on the type and degree of impairment, which aims to achieve a more pleasant and safer everyday life. Based on the displayed interaction between all elements, it is possible to provide the user with all information from the home environment, information about the user's health (pulse, temperature, pressure, sugar, etc.), and the possibility of calling the emergency services or the caregiver in case of incident situations. Management of sensors, actuators, and devices in the home is possible through speech using a virtual assistant, which improves the mood of the user in his home, if it is about person with impaired vision or person with physical disabilities. person with voice-speech difficulties and person with hearing impairment must receive adapted video-visual information to also provide them with a higher quality of life and independence.

Solution described in this paper is an assistive technology model that includes a device, user, service, and environment, providing a personalized smart home experience. With relevant devices and sensors gathering specific data sets, defined by a mathematical model, this solution serves as a vital sub-log for potential application of AI technologies. Laboratory testing has confirmed the significance of defining data sets in generating user-friendly information to the end user. In future development, an API interface should be defined for the sensor to segment and merge data based on user requirements. Future research will be focused on the development of ML models, verification of model effectiveness (simulation and validation), and the creation of a user knowledge base, all to model new IoT services for predictive user information.