An Integrated Support System for People with Intellectual Disability

1. Introduction

Intellectual Disability (ID) is one of the commonest neurodevelopmental disabilities worldwide and, in 2019, was estimated to affect about 2% of the global population, with significant regional inequalities (highest prevalence rates in low-middle sociodemographic index regions) [1]. Although ID may be isolated, it is frequently linked to other neurodevelopmental disorders, such as autism, sensory or motor impairments (hearing or vision), sleep and eating disorders, and medical conditions like epilepsy. It can also be associated with a wide range of psychopathologic issues, such as anxiety, depression, and emotional regulation disorders [2]. Having an ID condition can affect individuals in significant ways throughout their entire lives. It can impact important aspects such as health, well-being, education, recreation, employment, citizenship, community participation, and financial stability [1]. People with ID are one of the most vulnerable groups in our society that deserve a great attention in order to deal with difficulties in managing daily activities and accessing health services, but also to participate more actively in education, creative avocation, recreation, and to enhance the interaction with other people. Addressing these issues could promote their independence, self-confidence, self-sufficiency, stay healthy and inclusion in socio-cultural activities.

Assistive technology (AT) could be a vehicle for better self-functioning and independence and for promoting the overall well-being of people with ID, but also, as highlighted by WHO, for improving population health and upholding basic human rights [3]. Boot et al. in [3] refer to the factors related to the use of assistive technology by people with IDs; they categorize them in two groups: a) aspects of ID that may be associated with need for AT including intrinsic aspects to ID (impairment of adaptive or cognitive functioning), comorbidities (sensory, motor or neurological (e.g. dementia, epilepsy) impairments, mental health problems, obesity, etc.), multimorbidity, frailty, misdiagnosis, underdiagnosis, and common impairments, and b) aspects of AT provision that may be associated with ID including AT for ID (e.g. simplified mobile phones), AT for other impairments often associated with ID (e.g. hearing aids) and AT for other impairments (e.g. prosthetic limbs). In recent years there has been a growing effort to offer assistive technology for people with special needs including people with intellectual disabilities. To be successfully adopted by people with ID, it is crucial to identify and address any barriers that may hinder their usage. AT has been recommended as an intervention to enhance the quality of life for patients and their caregivers in the United Kingdom’s clinical practice guidelines [4]. Regarding comorbidities of people with ID, AT has the potential to increase the safety, confidence, and independence of dementia patients, while also reducing behavioral and psychological symptoms and maintaining cognitive and social functioning [5,6,7,8].

Along the same direction, a recent study [9] concluded that using digital devices can have positive effects on children and adolescents with ID, including improvements in executive function, basic cognitive-linguistic skills, academic skills, and social and behavioral skills. A detailed analysis of the characteristics that a content recommendation system should have in order to benefit individuals with ID, is reported in [10]. On this concept, we present an integrated system, which consists of several components performing different functionalities tailored to the special need of people with ID. The proposed system aims to deliver an appropriate, interactive content recommendation platform for people with mild, moderate and severe ID, while additionally, serving the detection of emergency cases. The latter concern the current health status of people with ID through monitoring of critical biosignals by means of a smartwatch and the disorientation and/or wandering situations, through location monitoring by a smartphone [11]. Moreover, the system provides appropriate alerts to the caregivers with respect to the detected urgency cases. In addition, since the proposed platform integrates automated speech recognition and text to speech technologies on smartphone, it offers information, entertainment and communication services through vocal requests and response of a virtual assistant, through a specialized smartphone application, which also includes the automated assistance of the system in emergency circumstances. In addition, other notifications, such as medication reminders, towards users with ID, their carers and close relatives have been integrated in the platform. Finally, the smartphone is equipped with an emergency (S.O.S.) button for voluntarily usage when the individual with ID feels uncomfortable.

The proposed platform constitutes an easy-to-use solution, serving information and entertainment, education and communication, but also providing support and assistance in urgency situations related to health issues, disorientation, or other inconvenient situations [12]. Furthermore, the proposed platform takes into consideration the crucial issue of exchanging information in a secure way, concerning the protection of personal data, but also the safety of the content that people with ID have access to. The latter is achieved by pre-filtering the candidate recommendations in order to exclude violent, or any kind of potentially harmful content.

Out system has been designed and being developed within the framework of QuaLiSID research project. Its novelty lies in the fact that it combines all the above-mentioned characteristics to an integrated platform tailored to the needs of people with ID, regarding the user-interface, the content recommendation services for information and entertainment, but also the automatic detection of emergency cases. The architecture of the proposed platform is depicted in Figure 1.

2. Related Work

Several initiatives, research projects, applications and platforms have been developed revolving around the use of digital technologies, aiming to assist Individuals with ID (IID).

In the training and educational domain ELPIDA e-learning platform [13] aims at providing training to IID, as well as awareness raising, and/or attitude change in the areas of human rights, communication, stress management, transition to adulthood, sexual health and aging. The ENABLE inclusive learning platform [14] offers training services to IID, their families, professionals in ID and local community representatives, while Stomp interactive platform [15] has been designed to support social and physical interaction for IID, through tangible user interfaces. Moreover, KIDEA [16] constitutes an innovative application that provides multimedia content for learning fundamental life skills, in order to increase autonomy and independence and the MAS platform [17] aims to assist and reinforce the learning capabilities of IID, offering adaptive games and data processing / monitoring tools. IDPLIVING e-training platform [18] supports the understanding of educational material and experiential educational activities for IID, aiming to strengthen and facilitate their community life. Moreover, the LudoMinga games-based platform [19], has been designed to facilitate the learning process of people with IDs. The platform is focused on providing an inclusive and accessible learning environment. Following this, Healthy Mind [20] is an online Easy Read tool designed to help people with IDs to recognize and regulate their thoughts and feelings.

On the socialization domain, the Greek social networking platform DisabledBook [21], offers to individuals with physical and intellectual disabilities free multimedia content, such as movies and music from disabled radio producers. Furthermore, MySigns web application [22] can be used to support mental health assessment of IID and significant communication difficulties. This tool is designed for collaborative use by individuals with intellectual impairments, their caregivers, and their mental health clinicians. Finally, regarding the a complicated issue of employment for IID, technical efforts such ERGASIAMOU job search platform for people with intellectual disabilities in Greece [23] and Inclusion International network [24] attempt to enable individuals with disabilities, to find employment and earn a - as possible - financially independent living.

Furthermore, several web-sites have been constructed in order to support IID and their families, such as the Puzzle web-site [25] aims to facilitate access to information, education and training for young people and adults with ID in some European countries. The epic web-page [26], features the services of a non-profit disability support provider specializing in essential services, such as transportation, home care, physical activities, community living support, etc., that promote independence. Other web-pages, e.g. the site of the DEP project [27] is addressed to IID and their families, who are in need of psychological support and rehabilitation services.

Regarding the recommendation services to IID, there is a limited number of projects dealing with the recommendation of multimedia content [10]. The delivery of personalized multimedia content in interactive Internet Protocol (IP) television environments in [28] is carried out by taking advantage of information gathered through the users’ interaction with the system and generating profiles that reflect groups of similar users, while it consequently adapts the group profile to each user’s preferences. Moreover, the smartphone-based MUBS recommender system for behavioral activation [29] introduced a personalized content-based activity recommendation model using a unique list of 384 enjoyable activities, which are designated with a default difficulty level and classified into one of the following six categories: work and education, daily living, practical, spare time, movement and social.

Moreover, research suggests that in many cases health problems of IID are either undetected, not communicated to the caregivers, or poorly managed by healthcare services [30]. Recent advances in ICT and particularly IoT/wearable technologies [31] can contribute significantly by offering early warning for different symptoms, thus improving the well-being and quality of life of IID and their caregivers [32]. For example, Dibia [33] developed a smart watch application to help users improve their ability to focus on tasks, reduce their anxiety via mindful meditation and improve their overall mental health via positive message priming. Recent smartwatch applications have also been aimed to detect Covid symptoms exploiting heart rate variability (HRV) [34]. In more detail, current smart watches in the market can support many different sensors that can be used for health monitoring (e.g., heart rate, respiration rate) [35], including specialized sensors for health monitoring, such as ECG sensor. However, up to this day most smart watches encounter problems in properly measuring very common biomarkers, such as body temperature [36].

Additionally, disorientation and wandering is also one common behavior of people with ID that might occur in their daily activities. This behavior is described by research on Autism Spectrum Disorder (ASD), as movement through space lacking intention or exact destination, as when a person is disoriented or not self-aware [37]. Wandering away from home or facilities is dangerous for IID and stressful for families and caregivers when those who go missing cannot be located. Frequently, wandering activity is associated with adverse events such as falling, elopement, getting lost, and emotional distress [38]. In this context, emerging technologies offer a promise of comfort and security in being able to easily locate a missing one. Therefore, a lot of research has been dedicated to the detection of disorientation or wandering-like behaviors, a complex task highly influenced by the location-based technologies used and the context in which people move. Martino-Saltzmanet al. [39] proposed one of the most widely accepted classification of wandering movements, defining three differentiations: (a) pacing: moving back and forth between two points, (b) circular wandering: circular movements between, at least, three points, and (c) random wandering: an unpredictable and inefficient course without repeating points. Algase [40] suggested wandering movement as a spatiotemporal locomotion behavioral pattern that consists of rhythmical movements consisting of two phases: walking and non-walking, and before each non-walking phase, one of the aforementioned walking patterns is presented. Two main research objectives pertain to wandering of populations with cognitive impairments. The first focuses on wandering evaluation based on offline analysis of trajectory data. For example, in [38] an algorithm is suggested to detect Martino-Saltzman et al. [39] wandering movement spatial patterns by analyzing the direction alterations in trajectories. Furthermore, in [41] a framework is presented that allows the classification of GPS-based mobility indicators commonly used in literature based on several characteristic and analytical aspects of mobility. The second one focuses on wandering detection by designing, assistive systems to provide IID safety assurance based on online observations. For example, in [42] authors developed an autonomous GPS system that monitors the location of patients, looking for abnormal users’ behaviors like when the person: (i) is entering in predefined dangerous zones, (ii) has exceeded a certain speed, (iii) has stopped moving for a specific time or, (iv) is wandering. Lin et al. [43] focused on real-time detection of spatial wandering patterns, i.e. pacing and lapping movements from users’ GPS traces, by examining count turning points in each on going trajectory. However, wandering patterns may vary for different people, hence, there is no widely accepted technique to automatically detect wandering behavior.

Regarding the mobile devices, including mobile phones, smartphones, and tablets, are highly accessible forms of AT and are more widely used than personal computers and older portable electronic devices [44,45,46,47,48]. Smartphones and tablets provide adults with intellectual impairments a chance to access social media platforms, enabling them to connect, communicate and participate with others, which can help alleviate the negative effects of social exclusion and stigma [49]. The smartphone revolution has transcended age barriers, extending its impact beyond just children and young adults to also include older adults. The ownership of smartphones among older adults has shown substantial growth, rising from a mere 10% in 2011 to an impressive 61% in 2021 [50].

One of the significant advantages of smartphone and tablet technology is its suitability for healthcare interventions. These devices come equipped with various features such as Internet access, mobile telecommunications, sensors, geolocation data, and notifications. Additionally, they offer the capability to install clinically focused applications [51]. With built-in sensors, smartphones and tablets can provide support similar to dedicated assistive devices without the need to carry an additional device constantly or the potential stigma associated with more visible assistive devices [52].

Furthermore, as these technologies become increasingly integrated into everyday life, they are likely to be familiar to users. This familiarity makes them easier to adopt as assistive technologies for older adults [5,8]. Despite the widespread adoption of smartphones and tablets in modern society, rehabilitation practice has been slow in embracing these new technologies [44]. In a recent study, Conte et al. [53] present a mobile-based augmentative and alternative communication solution designed as a tool to aid the rehabilitation process of people with ID. Some of the smartphone applications associated with ID or its comorbidities are listed below:

• Soy Cappaz [54] is a mobile application for IID, available on Android worldwide in English and Spanish. It displays an opening screen with four main sections: a) My Calendar, b) Where Am I?, c) My Tasks, and d) I Need Help. The application helps on work tasks, answers questions, and provides guidance. It’s also useful for daily activities like using a microwave or catching a bus.
• Viamigo [55,56] is a digital travel assistant with a mobile application interface for IID. It aims to promote independent mobility while reducing the burden on informal caregivers. It teaches users specific routes they can navigate independently, with the informal caregiver monitoring them.
• MindMate [57] is an application designed to support individuals living with dementia and their caregivers. It offers features, such as cognitive exercises, reminders, and mood tracking. The application also provides access to articles and information related to dementia care.
• MemoryWell [58] is an application that helps create life stories for individuals with dementia. Caregivers and family members can use the application to build personalized narratives and share them with healthcare professionals to improve patient-centered care.
• Alzheimer’s Society’s Talking Point [59] is an online community and mobile application that allows people affected by dementia to connect, share their experiences, and seek support from others in similar situations.
• Elder 411 [60] is an application designed to provide practical advice and tips for caregivers of individuals with dementia. It covers a wide range of topics, including communication strategies, safety measures, and resources for additional support.
• Timeless [61] is an application that offers reminiscence therapy to people with dementia. It allows users to access a vast library of pictures, music, and videos from the past, which can stimulate memories and encourage conversations.
• CogniCare [62] is an application designed to help caregivers manage their daily responsibilities effectively. It provides tools for medication tracking, appointment reminders, and communication with other caregivers or family members.
• GPS SmartSole [63] has been designed for caregivers to monitor the location of individuals with dementia who wear a special shoe insole with a built-in GPS tracking system. It can be particularly useful in cases where the person with dementia tends to wander.
• PainChek [64] is an application designed to help healthcare professionals assess pain levels in individuals who may have difficulty communicating, such as those with advanced dementia.
• Puzzle with Me [65] allows caregivers and their loved ones with dementia to solve puzzles together virtually, fostering a sense of engagement and connection.

3. Participating Users

Individual with ID that participated in our study are supported by a center of Creative Activities in Greece, that offers professional support to people with special needs. In order to protect their privacy, the center will be referred to as “AC”. A multidisciplinary team (psychologists, social workers, gymnasts, musicians, support staff, etc) at AC work together to enhance the creativity and daily functioning of individuals with disabilities. The center offers a wide range of programs, including educational activities such as music-kinetics, gymnastics, gardening, art, painting, computer skills, cooking, and games, as well as daily life activities like eating, personal hygiene, and housework. Additionally, AC provides social interaction opportunities that can greatly improve the skills and quality of life of people with disabilities. The participants are seventeen adults (six females and eleven males), all 21 years old or older and having varying degrees of severity, categorized as mild (three), moderate (three), severe (nine) [66], with one participant’s classification be unknown.

The participants possess various abilities and skills. They have varying degrees of autonomy (12.5% very good, 25% good), perception (6.3% very good, 25% good), cognitive abilities (6.3% very good, 31.3% good), ability to concentrate (12.5% very good, 6.3% good), memory (12.5% very good, 6.3% good), emotional maturity (12.5% very good, 18.8% good), motility (18.8% very good, 43.8% good), stereotypic movements (25% good, 25% moderate), cooperation ability (6.3% very good, 12.5% good), and participation in individual (6.3% very good, 25% good) or group activities (6.3% very good, 18.8% good). Additionally, they can read (18.8% very good, 12.5% good), write (18.8% very good, 6.3% good grade), speak (12.5% very good, 31.3% good), and use everyday vocabulary (12.5% very good, 31.3% good). In terms of self-care, some have good skills (37.5%), while others have self-care abilities (43.8%). When it comes to work, some have semi-skilled (6.3% good) or unskilled work with (18.8% very good, 25% good) or without supervision (18.8% good), while others do household chores (25% good) or receive assistance with housework (25% very good, 31.3% good). Overall, their individual abilities and skills vary greatly.

Due to the specific characteristics of the people with ID, the ethical component, privacy and data protection policy are ensured and considered a priority. Before proceeding with any research involving IID, we made sure to obtain informed (written) consent from their legal guardians, who are typically their family members or next-of-kin. This consent confirms that they agreed to allow the person under their care to participate in the study. Approval to conduct the study was obtained from the Ethical Committee of the Technical University of Crete. In the study, ethical principles were followed in accordance with the Helsinki Declarations [67].

Fifteen IID have been interviewed by specialists using a semi-structured questionnaire to gather information about their preferences in various categories such as entertainment, creative avocation (hobbies), news and information, virtual navigation, interactive games, and educational activities. These interviews aimed to gather information and create user profiles for the proposed system. Additionally, the findings aided in developing scenarios for the pilot study of the proposed platform. The questionnaire is comprised of three parts. The first part consists of ten questions that ask users for their personal information, such as their sex and age range, as well as their computer usage habits. The second part of the questionnaire consists of six questions about the six general categories and their subcategories. The third and final part includes three questions that measure the user’s willingness to use the web application in the future, their requirement for computer usage assistance, and their interest in a hypothetical new category that was not covered in the given categories. Users are asked to rate their agreement or disagreement with positive and negative statements about the choices provided by the platform using a three-point Likert scale.

In addition, five narrative interviews were carried out with individual staff members to gather data regarding their experiences with communication, daily routine, activities and special needs of IID from the point of view of their profession.

As mentioned, the platform provides modules for people with ID, their families, and caregivers in occupational activity centers, learning support centers, and other residential environments. In order to ensure the platform’s effectiveness, a short-scale pilot study was conducted in order to test the functionality of the subsystems and the integrated platform prior to the final evaluation and validation. The study concerns the real-time testing of the web-based recommendation services, as well as the smartwatch and smartphone applications through the components usage by the IID. The experimental testing was conducted in the AC environment with the IID users of mild, moderate and severe ID level.

It is important to note that the research team interacted directly with the people with ID who participated in this study, along with the continuous support of centre specialists (psychologist, occupational therapist, etc.) during the development, testing and evaluation of the proposed platform. Parents were not present during interviews and pilot study.

4. Content Recommendation Functionality and User Interface

Content-based systems are often used to deliver entertainment, educational or informational material through web applications and corresponding mobile devices [68,69]. Content-based filtering technique, which is utilized in the proposed recommendation system, analyzes the new content and calculates the similarity of items [70], aiming to recommend items similar to those that the user has selected / consumed in the past. In order to perform personalized recommendations, the user preferences, are stored in the profile and are constantly updated based on explicit or implicit user feedback. In these cases specific similarity measures, such as cosine similarity, can be used to match the new content to the user’s current preferences. Following this, the most relevant material is suggested and displayed in the priority order resulting from a ranking process based on the highest similarity between each element and the user profile [71].

Leveraging this concept, in order to generate web-content recommendations to the users, the semantically relevant items are identified, based on the information stored in the individuals’ user profile and expresses their preferences, interests, and special skills. The user-friendly interface is designed for convenient use by people with ID, who encounter difficulties in efficiently handling Information Technology (IT) systems. The detailed thematic categories, where the content is classified, have been formed according to the general preferences and interests of people with ID, following their responses on interviews and questionnaires, described in Section 3. Additionally, the individual preferences in the user profile are automatically updated according to the actual user’s web-activity (implicit feedback), which reduces the manual involvement of of people with ID, allowing to constantly receive suitable recommendations.

4.1. Backend Functionality

The content recommendation services are provided through a user friendly web-based interface in cooperation with the system’s relational database. The latter has been developed in MySQL, in order to store the users’, content’s and system’s data, as well as facilitate the platform’s functionalities.

Apart from the system administrator, the users are distinguished by main groups of:

• People with ID (adults and children), who are the key users of the recommendation system and may use it directly themselves, or with the aid of their relatives or/and caregivers;
• Their close relatives, i.e. parents and family, who have access to view their personal data registered to the system;
• The caregivers that represent the professional carers - occupational therapists who are allowed to access and modify the preferences, location, biomedical and medication data of the people with ID that they have under their provision.

Through the proposed system, people with ID have access to the pre-analyzed online material that falls under the six main categories of News, Entertainment, Education, Creative Activities, Interactive Games and Virtual Exploration, which contain several detailed categories, depicted in Table 1. Each of them consist of sets of particular descriptive human-interpretable terms (keywords), namely the textitprototype terms that actually represent more detailed categories’ aspects. For instance, the category of “Cities” contains “building”, “square”, “street”, “center”, “cathedral”, “monument”, “transportation”, “population”, etc. Moreover, each prototype term is accompanied with a degree of relevance, namely the textitprototype weight, corresponding to its association with the current category.

The corresponding items are published on a daily basis on secure websites and are of multiple formats, including text, image, video and multimedia. Due to the limited number of websites that are especially addressed to people with ID, the recommended content is mainly selected from general purpose web-pages, where multiple items are available, such as Photodentro [72], Openbook [73], Travel All Over Greece [74], Pixabay [75], Pexels [76], YouTube [77], Google Arts & Culture [78], Wikimedia Commons [79], Airpano [80], etc. However, the suggested items have not been stored in the proposed system’s server, but rather, the selection of a specific title directly leads to the web-source of the content, where it is publicly available.

The online content of multiple formats, which is recommended to the users is initially semantically analyzed and classified into one or more detailed thematic categories. This analysis leads to the extraction of metadata including the most descriptive and frequent terms contained in each item, namely the metadata terms. Additionally, the weights related to the terms’ frequency of occurrence of the metadata terms in the items’ content, i.e., the metadata weights are also specified.

The user profile consists of the entire set of the thematic categories along with the corresponding degrees of preference, ranging between 0 and 1 and consists of three levels: Low [0–0.3], Medium (0.3–0.7) and High [0,7–1] [81]. The detailed preferences in the user profile are initialized based on explicit user feedback, namely by the information that has been gathered through questionnaires and interviews of the participating users, namely the people with ID. Additionally, the prototype terms are also included in the user profile, along with the respective weights, which differ among users. The latter are initially calculated from the user-defined preference inherited by the parent category, multiplied with the respective prototype weight, and express the intra-category and inter-user differentiation.

4.2. Content Semantic Analysis

Aiming to support content recommendation by estimating the relevance of media items to the thematic categories of the platform, we designed methods for image, video, and text semantic analysis. Concerning the image semantic analysis, we exploited three publicly available models pre-trained on the ImageNet [82], Places365 [83], and YouTube8M [84] datasets (1000, 365, 3886 semantic labels, respectively), as well as two new models that we trained on the TRECVID SIN [85] and the Kaggle Sports100 [86] datasets (300 and 100 semantic labels, respectively). These five models constitute model set I and can annotate images with more than 5000 unique semantic labels. In order to semantically analyze video, we first segment the video into shots using the method of [87]. Then, leveraging the model set I, we analyze three key-frames per shot (selected by temporally-uniform sampling frames within the shot’s duration), annotating the shot with the most confident annotations of all key-frames. Additionally, we adopted the method of [88] and modified it for the annotation of each video shot with event/activity labels of the MiniKinetics [89], and ActivityNet [90] datasets (both with 200 semantic labels) with the two resulting models, constituting model set V. Our modifications on [88] concern dropping the object-level processing for the sake of computational efficiency. Regarding the text semantic analysis, we encode textual items in the joint feature space of [91] where we can compute the similarity to the - encoded in the same space - thematic categories’ labels.

We designed two approaches to estimate the relevance of a media item to a thematic category. In the first approach, categories2concepts, we initially use the Sentence-BERT text encoding method [92] to match each semantic label supported by I and V model sets, to the labels of the thematic categories. Through a manual filtering procedure, we selected the most relevant from the top 100 matches. When a media item is submitted for analysis, it is first annotated with semantic labels from the appropriate model set (i.e., I for images, I and V for videos). The relevance of a media item to a thematic category is relative to the confidence scores of the matched semantic labels.

Specifically, to calculate the semantic relevance of an image (or video’s shot key-frame) s to a thematic category q according to the categories2concepts approach, we follow the steps below:

• As a preparatory step, we construct the set of lists $C_i^{concepts}$ , $i=1,...,52$ with each list containing the semantic labels of sets I (and V), that are matched to the i-th thematic category
• We proceed to the semantic analysis of the image (or video’s shot key-frame), inferring a probability score for each label of the I (and V) model set
• Assuming there are n models in I (and V) model set, for each model, we sort the labels in descending order, based on their inferred probability, into the lists $R_j$, $j=1,...,n$
• We initialize a set $D=\varnothing$
• For each label in the $C_i^{concepts}$ list, we find its position in the $R_j$ list, and append to the set D ($\left|D\right|=\left|C\right|$) the normalized value of the position (i.e., position in the ranked list to the total number of concepts supported by the j concept set)
• The semantic relevance of an image (or video’s shot) s to a thematic category q is inversely proportional to the mean value of D, i.e. $si{m}_{concepts}(s,q)=1-\frac{\sum D}{\left|D\right|}$.

In our second approach, categories2avs, we start by constructing a pool of text sentences by aggregating the textual part of the four video captioning datasets i.e., MSR-VTT [93], TGIF [94], ActivityNet [90] and Vatex [95] datasets. Utilizing the Sentence-BERT text encoding method, we measure the text similarity between the sentences of this pool and the thematic category labels. After visually inspecting the top 100 similar sentences, we select the most relevant matches. When a media item is submitted for analysis, we calculate the cosine similarity between the item’s embedding and the relevant sentences’ embeddings in the joint feature space of [91]. The relevance of an item to a category is expressed as the maximum similarity of all selected relevant sentences. In summary, to calculate the similarity of an image (or video’s shot key-frame) s to a thematic category q according to the categories2avs approach, we follow the steps below:

• As a preparatory step, we have constructed the set of lists $C_i^{avs}$, $i=1,...,40$, with each list containing the items from the pool of sentences to which the label of thematic category q was assigned
• As part of the same preparatory stage, we have represented in the joint feature space of [91] the $C_i^{avs}$ set (i.e. the elements from the pool of sentences with which the thematic categories have been augmented)
• We compute the representation of s in the same joint feature space of the method [91]
• We initialize a set $D=\varnothing$
• We calculate the cosine similarity between the vector of the feature frame representation of the plane s and the vectors of the representations of each element of $C_i^{avs}$ and append the resulting similarity values to the set D
• The similarity of the image (or video’s shot key-frame) s to the thematic category q is expressed as the maximum value of the set D, i.e. $si{m}_{avs}(s,i)=max\left(D\right)$

To evaluate the two approaches, we employed the V3C1 subset of the TRECVID AVS dataset [96] and treated each thematic category label as a query to retrieve related V3C1 videos. Then, we performed a visual inspection of the top 50 videos returned by each approach, noting whether each video is relevant to the query. We used precision as the evaluation measure (i.e., correct answers against the number of examined videos, averaged over all categories). We observed that late fusing the results of categories2concepts and categories2avs approaches (by considering the maximum score from the two approaches), yielded the best results (62.8% versus 54.4% and 58.8%, respectively), therefore this fusion of both approaches was employed for estimating the relevance of a media item to a thematic category.

Moreover, the human-interpretable stem-vectors, as discussed in Section 4.1 were derived through the analysis of large number of media items found on the web. This set of prototype terms semantically describe each thematic category. The semantic analysis methods also calculate the relevance to each of these prototype terms (i.e., a similarity value in $[0,1]$) leveraging the matches of semantic labels to thematic category labels, computed for the categories2concepts approach. Based on the latter process the metadata terms are extracted.

The semantic analysis methods were deployed as a REST service with an endpoint for each supported media item type (i.e., video, image and text). In Figure 2 an overview of the semantic analysis process is illustrated while in Figure 3 the input media item (left side) along with an indicative selection of the results (right side) is presented. Specifically, (a) for a video we showcase the top-3 inferred thematic categories, as well as the top-5 concepts of the “imagenet” and “YouTube8M” concept pools, (b) for an image we showcase the top-3 inferred thematic categories, as well as the top-5 concepts of the “SIN” and “Sports100” concept pools, c) for a short piece of text we showcase the top-3 inferred thematic categories.

4.3. Inference Model

The inference model is the actual learning mechanism of the recommendation system being responsible for a) the selection of personalized recommendations regarding each detailed category and ranking based on the user preferences, b) the update of the preference degrees for each category included in the profile, based on the user’s current implicit feedback, i.e., the web-activity.

The inference model has been developed using the Python programming language and integrated with the back-end for retrieving information from the system’s database. It comprises of three scheduled tasks for updating the recommended content, performing user profile adaptions on prototype terms and content categories.

4.3.1. Personalized recommendations

In order to determine the particular online content for personalized presentation to the users, the inference model takes as input: a) the metadata extracted from the semantic analysis of the web-items, i.e., the categories where the items have been classified, the metadata terms and the respective metadata weights as described in Section 4.1, b) the users’ degrees of preference for each category and c) the prototype weights of each category stored in the user profiles. Firstly, the new classified content and the respective metadata are specified. Following this, the common terms are detected between the current item (metadata terms) and its categories (prototype terms) and the Cosine Similarity measure is calculated according to the following formula:

$$CoSim(U_P,M_W)=\frac{\overrightarrow{U_P}\ast \overrightarrow{M_W}}{|\overrightarrow{U_P}|·|\overrightarrow{M_W}|}$$

(1)

where $CoSim$ is the Cosine Similarity measure, $U_P$ stands for the vector of the user’s prototype weights of the common terms (between the content and the classification category) and $M_W$ represents the vector of the metadata weights extracted from the current content.

The results of the cosine similarity measure are used to rank the recommendations in an intra-category priority order. However, the user has the option of sorting the suggested content by its type (image, video, etc). Furthermore, depending on the user’s degree of preference in each category, the content is recommended according to the following rules: a) all of the most recent items for the High preference categories, b) up to 10 items for the Medium preference categories and c) up to 5 items for the Low preference categories.

4.3.2. User profile update

In order to dynamically update the categories’ degrees of preference in the user profile based on the recent user online activity, all the prototype weights of the terms contained in the individual user profiles should be adapted. The prototype weights are constantly updated using the following input data: a) the metadata of the web items that were recently selected / consumed by the user, b) the metadata of the suggested web items that were ignored by the user and c) the user prototype weights of the categories where the suggested content is classified. The updated prototype weights of each category in the user profile result from the mathematical formula 2, which is based on the approach [71] and has been adapted to the particular characteristics of the proposed platform. More specifically, it has been modified taking into consideration the main differences and individual features of our system, compared with the system in [71], namely:

• The approach of [71] deals with a mobile-phone application, where the user is limited to a single screen, which does not constitute a web-based interface.
• The content concerns textual news items in [71], as opposed to our system, where the content is multi-modal and derived from several different sources.
• The most important difference concerns the target users, which, in our case, constitute IID that are characterized by special behaviors, e.g., they might have opened a webpage, for a long time, without being concentrated on the content itself.

  $$W_{new}=W_{old}\pm M{W}_t·e^{-\beta ·U_b·U_h}$$

  (2)

  where $W_{new}$, $W_{old}$ are the new and the current prototype weight respectively in the user profile, $M{W}_t$ corresponds to the average of the metadata weights of the specific term in the entire set of items presented to the user. The weights of a specific term in the user-ignored items are subtracted from those contained in the selected ones. Hence, the +or - sign is applied where the metadata weights of the term prevail concerning the consumed or ignored items respectively. Moreover, $e^x$ is used to follow the personalized nonlinear change of the prototype weight with respect to the usage term’s history. The changing rate of the weight is inversely proportional to the value of the parameter x, where $U_h$ stands for the number of the selected items, where the term exists and $U_b$ represents the indicative mean number of the daily selected items, computed every week, i.e., the more items a user consumed per day, the more slowly the prototype weights increase in the profile. Furthermore, the $\beta$ constant is used to differentiate between the changing rate of the weight if the update is performed concerning an interesting, or an ignored item, taking different values for positive and negative user feedback. More specifically, in the case of ignored items, the changing (decreasing) rate should be slower since a non-selected item does not constitute an explicit indication for non-interest. For instance, it can be interpreted as already read from another source, or as possible that the user had no time to spend on it. On the contrary, in the case of consumed items the changing (increasing) rate should be faster since a selected item demonstrates a strong indication for interest [71]. Based on the numerical values resulted by applying the formula, in Equation 2, the indicative values for the $\beta$ constant have been set to $\beta =0.01$ for selected items (positive feedback) and $\beta =0.02$ for non-selected items (negative feedback). Note that the weight adaptation concerns only the common terms between the user profile and the currently recommended items.

Finally, each new category’s degree of preference is calculated as the average value of the included terms-weights, where each of them is divided with the respective initial prototype weight, i.e., the term’s degree of relevance:

$$P_{new}=\frac{{\sum }_{n=1}^{n=k}\frac{W_n}{R_{Pn}}}{k}$$

(3)

where $P_{new}$ is the new preference degree of the current detailed category, k is the number of the prototype terms associated to the category, $W_n$ stands for the prototype weight of each term and $R_{P}n$ corresponds to the relevance degree of the term with respect to the current category.

Subsequently, the new preferences of the first-level categories result from the average of their subcategories, as follows:

$$P_{Gnew}=\frac{{\sum }_{n=1}^{n=l}{P}_n}{l}$$

(4)

where $P_{Gnew}$ is the new degree of preference ofn the current general category, l is the number of the detailed categories associated to each general one and $P_n$ stands for the preference degree of each contained detailed category.

It should be noted that the categories’ degrees of preference are not updated as regularly as the prototype weights (daily), but gradually, in a longer-term basis (e.g., once per week).

4.4. Web-based User Interface

The web-based user interface is being constructed in the Greek and English language. The user enters the proposed platform through the login page, that allows the different user-groups to interact with the integrated system. More specifically, people with ID are allowed to view news, entertainment, etc, content, while their carers and close relatives are able to access the data and preference profiles of the people with ID under their support. Hense, the user interface contains four tabs corresponding to the page of a) login (and registration on new user), b) people with ID, c) caregivers and d) families (parents, close relatives, etc) of disabled people.

The images that illustrate each category, as well as the functionality icons have all been downloaded from Pixabay [75] and Pexels [76], where they are freely available and are not subjected to license issues.

The development of the user interface is based on the WordPress tool version 6.2 442 combined with rhe WampServer, which is a windows-based web development platform that enables creating robust web-applications with MySQL database and PHP Apache2 functionality. WordPress handles the user login and registration, and displays the web pages that are visible to the users depending on their role in the system. A RESTfull API using the Python library FastAPI (version 0.92.0) was developed for the interaction of the user interface with the database.

Communication of front-end and back-end modules is achieved via REST calls (GET, PUT, POST, DELETE requests) using the JSON format. All information is stored in the MySQL database on which the back-end module can connect to, and retrieve information for fulfilling the requests of the front-end.

4.4.1. Interface for People with ID

People with ID enter the system through the login page that, apart from the search option, includes six different content categories (Table I). The selection of the general categories leads to sets of more specific subjects, such as Music, Movies, Theater, etc, through which people with ID can access the listed recommended internet items. The structure and functionality of the User Interface addressed to people with ID is illustrated in Figure 4, while the suggested content is depicted in Figure 5.

After the selection of a sub-category, the user is led to a list of titles that correspond to the most recent web-items, which are recommended in appropriate ranking order, based on the preferences, interests and special characteristics reflected in the user profile. Additionally, the user has the option of sorting the content according to its type, as illustrated in Figure 5. However, since some categories contain limited content (e.g. Virtual Exploration), or material which is not regularly updated (e.g. Education), the recommended content may concern the entire category corpus and not only the recent items.

4.4.2. Caregivers’ and Families Interface

The web-interface of the proposed system allows caregivers and close relatives of people with ID to access their data that have been registered in the platform under personalized credentials.

When the caregivers login the platform they are automatically led to the corresponding user user interface that allows them access their personal profile, as well as the personal data of the people with ID who are under their provision (Figure 6). Apart from the personal data of people with ID, the carers are allowed to specify, view and modify the platform’s data, which are contained in their user profile. These data concern their medical history, their current medication, specific biomedical and location parameters monitored by the smartwatch and the smartphone’s GPS respectively, weekly statistical reports of health parameters and the health and disorientation related emergency-alerts history. More specifically, trough the web-interface the caregivers have access to the following data sources:

• The detailed preferences of the supported people, which the caregivers can anytime alter, in case they believe that the system automatic updates do not follow the actual individualized degrees of preference.
• The medical history including reports and medical imaging modalities, which the therapists are allowed to (optionally) store in the system’s database.
• The current medication of a specific benefiter with ID that the caregivers can store in the platform (Figure 7). In addition, through the button of ’Reminders’, they can specify, as well as modify when/if necessary and schedule the time intervals of the corresponding reminders for sending to the people under their provision.
• The current values of biomeasurements monitored by the smartwatch, i.e. the heart rate, the oxygen saturation, the body temperature and the stress level, which the carers can anytime view (Figure 8).
• The daily and weekly statistical reports of health parameters, namely specific statistical metrics, such as, mean, maximum and minimum values, etc, which are constantly calculated by the system by means of the received time-series of the heart rate, the oxygen saturation, the body temperature and the stress level (Figure 9).
• The current location parameters monitored by the the smartphone’s GPS, which the carers can anytime view. Additionally the caregivers are able to specify and change - directly on the map - the polygons that represent the individual safety areas of movement and activities for people with ID, namely geofences (Figure 10).
• The health and disorientation related emergency incidents that are automatically detected by the system along with the corresponding alerts. The carer can anytime view the recorded alerts, fill additional data, as well as register new emergency incidents. In Figure 8 the graphical environment of health-related alerts is depicted.

Regarding the closed relatives of people with ID, after login the system, they can view all the above-described data concerning their familiars, yet they are not permitted to change any of these.

5. Mobile Applications Functionality and Interfaces

5.1. Detection of Health Emergencies

Since IoT/wearable sensors can provide early warnings in emergency cases where people with ID are involved, the health monitoring service was integrated into the proposed platform in order to support the continuous measurement of specific biomedical parameters, i.e. heart rate, oxygen saturation, body temperature and stress level, providing the functionality of issuing appropriate alerts to the disabled users and their assigned carers in case the measured values exceed predefined normal ranges during the specific time interval of 5 minutes in our application. The latter have been defined according to the following values:

• 40 pulses/minute < Heart Rate < 100 pulses/minute
• 95% < Oxygen Saturation < 100%
• 35.7°C< Body Temperature < 37.2°C
• 0% < Stress Level < 75%

In this context, a market research was conducted in order to select smartwatches that include such sensors and support the development of smartwatch applications via the provided SDK, at least for Android mobile phones. Two popular platforms were identified, namely Garmin and Fitbit, which (a) support a wide range of smartwatch models with different prices and capabilities, catering to various user needs, and (b) provide adequate SDKs for supporting smartwatch application development. Our proposed smartwatch application (App) extends previous work described in [97], i.e. an open-source smartwatch application that was used to detect epileptic seizures and raise an alarm to warn the assigned carer that the user may need assistance. Specifically, two smartwatch applications were developed, with respect to each of the aforementioned platforms, and successfully tested with three mid-priced smartwatches, specifically the Garmin Vivoactive 4 and Venu 2 (Figure 11) and the Fitbit Sense device, to gather and send to a web server the physiological data, such as heart rate, and oxygen saturation measurements.

Regarding the body temperature parameter, although the majority of smartwatch models include a temperature sensor, it is typically used for ambient temperature measurement rather than body temperature. Moreover, since a smartwatch is usually worn on the wrist, body temperature measurement is unreliable due to the influence of perspiration etc. [98] and environmental factors. To meet this challenge, an effort has been made to improve body temperature estimation based on the use of the Kalman filter [99,100] using successive measurements of heart rate within the smartwatch application units. Given an available initial user’s body temperature, this model provides more accurate detection of body temperature fluctuations occurrences.

Furthermore, besides the provided aforementioned main bio-signals of heart rate and oxygen saturation and the generated body temperature, stress level feature was selected to be continuously monitored, since IID may be at a greater risk for experiencing high stress. User’s current stress level is determined on smartwatch based on their heart-rate variability; the lower variability between beats equals higher stress levels, whereas the increase in variability indicates decreased stress. The SDKs provide a stress level between 0 (resting phase) and 100 (high stress). Although stress is not designed to be a diagnostic marker, its detection can help users and their carers manage stress levels throughout the day and identify activities that cause tension; since research suggests that anxiety is a pertinent issue for adults with autism (that often coexists with ID), i.e., the more anxious the individuals, the less likely they are able to successfully cope with tasks [101].

Moreover, some further useful user activity data, provided by both the Gamin and Fitbit SDK, such as calories burned and steps counter parameters were selected for monitoring, since research suggests that impairments specific to autism, as well as general environmental factors, could lead to an imbalance between the intake and expenditure of energy, leading to obesity [102]. Therefore, monitoring of users’ total number of daily step count and calories burned, can be exploited for providing activity recommendations when a long inactivity period is detected.

All user data is stored in the MySQL database of the server using a REST API. More specifically, the sampling rate of heart-rate and oxygen saturation bio-signals is 1Hz, while average values are transmitted to the server per minute. Similarly, the estimation of body temperature and stress level value are updated every minute using the accumulated data of the past minute time window. Figure 12 presents an example of bio-signal and activity recorded data for a time duration of 12 minutes.

Furthermore, a tool for offline statistical analysis was developed, since such functionality was considered important for caregivers willing to study the previously captured data. This tool offers options for calculating statistics by day or week (e.g. mean, dispersion, minimum, maximum, kurtosis, skewness values, etc.) concerning the above-described biomedical parameters. As reported in Section 4.4.2, the statistical values (as well as the current bio-measurements) can be accessed by the caregivers and close relatives of IID, through the web-based user interface, accompanied by corresponding graphs, which can highlight time periods, where measurements are beyond the predefined normal ranges.

5.2. Detection of Disorientation and Wandering Behavior

The detection of disorientation and wandering behavior emergency cases have been integrated to the proposed platform. Initially, the caregiver specifies one or more closed polygons, which represent the safety zones for the IID, The definition of these areas, namely geofences, can be performed either through the web-based user interface (Figure 10), as reported in Section 4.4.2, or the mobile App interface as illustrated in Figure 13. Then, similar to the approach presented in [103], tracking is enabled, and when the user moves outside the predefined areas, a notification is sent to the caregiver.

The other case of alerts issuing concerns the so called "wandering" behavior. The design of the wandering-detection sub-system, combines best practices derived from recent research work, based on analysis of GPS trajectories recorded in the server. More specifically, users’ walking movements can be analyzed periodically to detect three wandering patterns according to the Martino-Salzman model of wandering behavior [39,40]. However, a drawback of this model is that wandering detection is based on analyzing sequences of movements between predefined locations, e.g. different rooms within a house or predefined locations in the city. Moreover, these sequences have to be segmented by defining specific rules (e.g., when the user remains static for N seconds), so this increases complexity for the carer. Thus, an additional simpler algorithm for detecting loops in GPS trajectories was also implemented based on the approach proposed in [104].

However, it should be noted that in the event of an emergency related to disorientation, the carers of the IID is able to activate (by pressing the "view" button) the display of the current location of the person under supervision, if they suspect that they may be lost, or that they may coping with a serious a health event (Figure 10).

5.3. Proposed Smartphone Application

People with intellectual disabilities often encounter challenges in acquiring reading skills. According to a comprehensive large-scale survey, reading difficulties were identified as the most common secondary condition associated with ID, with 67% of the participants reporting reading as a problem area [105,106]. Additionally, it is well-known that individuals with ID may also experience speech-related issues, such as slow pace, lack of articulation, and clarity, leading to difficulties in verbal communication.

Given the higher incidence of severe reading disabilities in this population, especially in relation to the intelligibility of natural speech, we have carefully selected natural, free speech as the primary means of interaction for our application. By leveraging automated speech recognition (ASR) and text-to-speech (TTS) technologies, we aim to provide a more accessible and inclusive experience for users with intellectual disabilities.

Moreover, to ensure ease of use and quick access to important features, such as calling for help or detecting health problems or disorientation, we have implemented automatic activation for these functionalities. In some cases, we have minimized the involvement of the user, allowing critical actions to be triggered with a simple press of a distinct button.

Through these thoughtful design choices, the proposed mobile application, namely QualiSID mobile App, aims to promote independence and enhance the overall user experience for individuals with intellectual disabilities, facilitating smoother interactions and better support in their daily lives. The QualiSID mobile App follows a modular development model, integrating and interconnecting all the individual implemented modules. During the application’s development the following infrastructures have been utilized, which are hosted on cloud servers:

• Apache Server [107]
• MariaDB database server [108]
• Node.js Server [109]

In terms of programming languages, the following were used:

• PHP [110]
• Javascript [111]
• HTML [112]
• CSS [113]

The proposed mobile application constitutes a Progressive Web App (PWA), that’s developed using web platform technologies, yet it delivers a user experience comparable to that of a platform-specific app, such as a native Android application. Like a website, a PWA is capable of functioning across various platforms and devices using a single codebase. Furthermore, it can be installed on a device, work offline and in the background, and seamlessly integrate with the device and other installed apps [114]. The application is multilingual, with Greek (full version) and English (extended version) being the first two languages that have been implemented. It consists of four main functional units, i.e., Support & Assistance, Emergencies & Alerts, Communication and Voice Controlled Virtual Assistant, which are presented in Figure 14.

The mobile application provides several services to the people with ID, but also to their caregivers, which are described as follows:

• People with ID:
  • Information, such as encyclopedia subjects, points of interests, weather forecast short answers, etc and entertainment, i.e. videos, pictures, music, radio, etc, through oral request (voice function and speech recognition) and response of the virtual assistant.
  • Communication with caregivers, relatives and friends by phone calls and sms messages through voice commands.
  • System support and assistance in emergency cases through automatic start of video calls and/or dialogue scripts and voice instructions with the virtual assistant.
  • Medication and other (such as lunchtimes) reminders by the virtual assistant.
  • S.O.S. (emergency) button for pressing in urgency situation related to health issues, disorientation, or other circumstances where the individual with ID feels uncomfortable.
• Caregivers:
  • Automated sound alerts for notification in emergency cases related to health issues that have been detected through the smartwatch monitoring of biomeasurements, i.e. heart rate, oxygen saturation, body temperature and stress level.
  • Automated sound alerts for notification in emergency cases related to disorientation and/or wandering behavior through the location monitoring from smartphone’s GPS, when the individuals with ID have been located out of the safety zones (geofences).
  • Specification of geofences which constitute the individualized safe activities areas for the supported IID, through the mobile user interface.
  • Specification of automated sending of medication reminders to the supported IID their carers and families through the mobile user interface.

In Table 2, the mapping of the individual modules to the main units of the QualiSID mobile application is depicted. Table 3 lists the main data required for the proper operation of the QualiSID mobile App and the achievement of its objectives. Furthermore Figure 15 presents the voice-controlled virtual assistant architecture, while the flow chart of the health parameters monitoring, location tracking and alerts related processes is depicted in Figure 16.

6. Experimental Results

In order to experimentally evaluate the effectiveness of the proposed system, 17 individuals with ID from the AC Creative Activities Center were involved in testing the developed components of the integrated platform.

6.1. Content recommendation service

A semi-structured questionnaire was constructed aiming to gather information for the preferences of IID in several detailed thematic categories, falling under the 6 general categories of News, Entertainment, Creative Activities (hobbies), Education, Virtual Exploration and Interactive Games (Table 1). Following this, the initialization of the detailed preferences in the participating users’ profile was performed according to the information collated through questionnaires and interviews of these users. In addition, based on the results of the completed questionnaires, the respondents identified 8 detailed categories as highly preferred, which include a) Timeliness & Weather-Forecast and Sports from the News general category), b) Animals & Plants, Music and Movies from the Entertainment general category, c) Internet & IT from the Education general category) and d) Cooking & Pastry, as well as Painting & Crafts from the Creative Activities general category.

In order to experimentally evaluate the effectiveness of the proposed recommendation service and the web-based user interface, a small-scale pilot study was carried out in real-time. To this end, 8 content sets were constructed, each consisting of 15 web items classified to each one of the above-mentioned detailed categories; namely 120 web-items in total were selected as recommendation material. The titles and source hyperlinks of these items are provided in the Supplementary Material document [115].

Concerning the testing process of the recommendation service though the web-based user interface, specific number of items were suggested to the users according to their initial preferences, particularly: 15 items for the High-, 10 items for the Medium-, and 5 items for the Low-preference categories. At first, the users ware asked to log in and subsequently select the preferred content items from each category. It should be stressed that during this pilot study, the users took their time in order to explore the content items at their pace.

Indicative results are presented for 3 end-users with different severity of disability, namely, mild, moderate and severe, as well as a brief description of the users’ peculiarities and problems regarding the use of the web-interface. In Table 4 the initial degrees of preference with respect to the categories are reported and in 5 the number and ID of the recommended content, corresponding to the initial preferences are depicted.

• User 1: This user has a mild ID. He understood the scope of the test and could use the web-based platform with help at the beginning. He performed the task well and was very concentrated during the experimental testing. The user selected items according to his original preferences (e.g. paintings & crafts) and other items not marked as High in the questionnaire (e.g. animals & plants) (Table 4 and Table 5). He used the system’s recommended ability to increase the font size of text elements to read the text more easily in the selected items. Based on his selected content items, the updated preferences in his profile were maintained in four categories and changed in the other four (reduced to three and increased to one). (Table 4).
• User 4: This user has a severe ID. He used the web-based platform with assistance. He performed the task pretty well during the pilot study. This user mostly selected items according to his initial preferences (Table 4 and Table 5). He can read to a minimal extent; he chose to look mostly at pictures and videos. In the beginning, was difficult for him to push buttons. He could not concentrate his attention and wanted to speak while performing the testing. Based on his selected content items, the updated preferences in his profile were maintained in two categories and changed in the other six (fell to six) (Table 4).
• User 12: This user has a moderate ID. She also used the web-based platform with assistance. She responded very well during the pilot study. She mostly selected items according to her original preferences (Table 4 and Table 5) and also chose to look mostly at pictures and videos. She is communicative; she could concentrate but also wanted to speak while performing the testing. Based on her selected content items, the updated preferences in her profile were maintained in five categories and changed in the other three (reduced to two and increased to one) (Table 4).

Finally, regarding the next system’s recommendation of the same content sets to the particular users, the number of the suggested items corresponds to the updated degrees of preference in categories, while additionally, the priority order of the items in each category has been determined according to the users’ feedback, i.e. corresponding to their previous choices. Therefore the previously selected content is ranked first, while the most relevant items are subsequently listed, as presented in Table 5. The results demonstrate that the updated degrees of preference as well as the ranking order of the new recommendations correspond to the implicit users’ feedback, i.e. to the previous choices during the testing process.

6.2. Health and location related alerts

Regarding the experimental testing of the health parameters monitoring, 4 smartwatches were used, paired with 4 smartphones, assumed to belong to the responsible caregivers. Each smartwatch has been worn by an individual with ID for 15 minutes, while the measurements parameters were performed per minute. In Figure 17 the indicative graphs of heart rate, oxygen saturation, body temperature and stress level time-series are illustrated for one1 of the respective 4-users’ groups with mild (1 user), moderate (1 user) and severe (2 users) ID.

As it can be clearly observed in the depicted measurements, all the parameter values lie in the predefined normal ranges, apart from the heart rate and the stress level of User 1, that have exceeded the normal limits for over 5 minutes. In this case corresponding notifications (alerts) were sent to the related carer’s mobile phone.

As far as the location tracking is concerned, a safety zone was defined covering the entire region of the AC center, i.e. the interior of the building along with the front and the back yards, depicted in Figure 13. Since none of the participating IID walked out of the boundaries of the geofence area, no disorientation alert was sent to the caregivers. However, this feature has extensively tested in hypothetical cases in order top verify its reliable functionality.

6.3. Smartphone application

The mobile application was evaluated based on its usability, user familiarity, and voice recognition accuracy. All but one users with mild, moderate, or severe ID were initially uncomfortable, especially with the virtual assistant, yet they eventually responded well to the test, after assistance by their carer. Three users of severe ID needed help to form some questions and ask for answers, whereas three others of mild and moderate ID were more concrete and specific to their questions. The user who responded very well during the testing posed many questions and had a very creative dialogue with the virtual assistant named Clio. The outcomes of the creative dialogues between the test users and the virtual assistant Clio were successful, with 70,45% positive answers to the users’ questions. It is noteworthy to mention that a large number of the “negative” answers, such as “I don’t understand you”, “I didn’t catch it” or “Rephrase, if you want”, and including three non-answers and two false positive answers, resulted from incomplete and missing questions, or due to the poor voice recognition with respect to the specific users with severe ID, whose utterance was not clear. In addition, some users seem not ready to dialogue with Clio. These results highlight the high usability of the proposed smartphone applications, a good level of voice recognition accuracy, and also the need to train the users to be more familiar with and have a creative dialogue with Clio.

7. Discussion and Conclusions

People with Intellectual Disability (ID) encounter several issues in communication with other people, interaction with their environment and the activities of daily living, since they have certain limitations in cognitive functioning and skills, as well as in social, mental and self-care abilities. To this end, technological advances can provide convenient interactive systems and services that address their special needs, in entertainment, education and creative activities, but also automatically detect issues that they are not able to sufficiently express. In this context, the Global Cooperation on Assistive Technology (GATE) program, initiated by the WHO, aims to enhance accessibility to affordable and high-quality assistive products for individuals with different illnesses, age-related conditions, and disabilities, including intellectual disabilities, which is a neglected area of research and practice [3].

In this paper the design of an integrated content recommendation and emergencies’ detection platform for people with ID is presented. The proposed system is being constructed on a web-interface that constitutes a user-friendly environment, which, based on its experimental evaluation, is fully handle-able and assisting for the users with mild, moderate and severe ID. It provides information and entertainment content, while suggesting educative and creative activities, according to the personalized preferences, interests and skills, reflected in the user profile. The latter is explicitly initialized, yet is dynamically updated through the inference model, taking into consideration the user’s online activity, that is recorded through the content’s selection. The recommended content constitute freely available web-items of various formats such as text, image, video, multimedia, for which the links to the source, rather than the files, are preserved to the platform’s server. In order to match the new available content with the current user preferences, the inference model exploits certain metadata, extracted using advanced AI-based multi-modal semantic analysis.

The rationale behind building such a recommendation platform is the inability of people with ID to efficiently handle and make use of IT systems. Hence, the automatic adaptation of the profile-stored preferences, according to the web-interaction of those people, allows them to constantly receive suitable recommendations, while it relieves them from the manual intervention on their individual preferences.

The proposed platform integrates additional functionalities, which serve the detection of emergency cases related to the health status of IID and the current area of movement and action. The health-status recognition is performed through the monitoring of specific biomeasurements, i.e. heart rate, body temperature, oxygen and stress level, carried out by a smartwatch. On the other hand, the location-related emergency concerns a potential disorientation which is detected based on the location monitoring by the GPS sensor of a smartphone, with respect to the defined user-specific area of physical activities. Furthermore, in emergency situations, the system provides automated notifications to the caregivers, in order to rush for help.

Moreover, the proposed system includes a specialized mobile application which provides information, entertainment and communication services to the IID upon vocal requests and response of a virtual assistant, through speech recognition and text to speech technologies implemented on smartphone, Automated assistance in emergency cases is also provided through the smartphone, which is additionally equipped with an S.O.S. button for voluntarily usage by the IID. Finally, medication reminders towards IID, their carers and family, as well as other useful notifications, related to the users’ daily routine are also supported.

In order to evaluate the proposed platform’s functionality, a short-scale pilot study was conducted, included experimental testings of the individual components in real-time and conditions. Seventeen users supported by the AC center of Creative Activities participated to the pilot study, starting from declaring their preferences concerning the detailed categories of our application, through dedicated questionnaires and interviews. Following this, the detailed preferences were registered to the individual user profiles. Considering the limited availability of content that caters to the unique requirements of IID, we have carefully chosen items with easily accessible content that can be viewed, searched, navigated, read, and interacted with. The content primarily consists of images, videos, audio and multimedia material, supplemented with text to engage users and cater to their preferences and needs. Our goal is also to motivate users to explore topics beyond their usual interests and provide them with appropriate stimuli. In practice, based on a closed question (giving the respondent a limited amount of options to choose from) from interviews, we can report that ten participating users expressed their initial interest in using the platform only for Entertainment, two users for news, two users for Creative Activities and one user for interactive games, while the Education and Virtual Exploration were not on their interests. On the other hand, based on the indicative results of the experimental testing but also the interaction we had with the participants in the AC center - a pleasant environment that matched very well with the design, implementation and scope of our research - we can claim that the users could explore, select and also enjoy with alternative topics. According to a study by Munir et al. [116], using MESE (Multimedia in Education for Special Education) - an assistive multimedia application involving different senses and media - improved reading skills and memory in children with ID. More specifically, regarding the experimental results of the recommendation service, our system’s had a high response, since the follow up content recommendations conformed to the previous user feedback, concerning a) the number of suggested items corresponding to the updated degree of preference for each category and b) the ranking order, of the items in each category, according to the detailed subjects of the users’ previous selections. The results demonstrate that the updated degrees of preference, as well as the ranking order of the new recommendations correspond to the implicit users’ feedback, i.e. to the previous choices during the testing process, verifying the high performance and reliability of the inference model.

The measurements of the health-parameters demonstrate that the detected values lie in the predefined normal ranges, which indicates their correct estimation by the selected smartwatch and/or improvement by means of the developed methods described in Section 5.1. Concerning one case where heart-rate and the stress level exceeded the normal limits for over 5 minutes, corresponding alerts were automatically sent to the carer’s mobile phone.

Moreover, regarding an additional biomedical measurement, namely the blood pressure, which was considered for inclusion to our application, this is mainly supported from inexpensive smartwatches. However, usually, (a) no support is provided for application development, (b) details about the measurement process are not provided, and (c) the reliability of measurements is unknown and possibly problematic. Greater reliability in blood pressure measurement is only reported for a few specialized but expensive watches [117], which still require a special measurement process by the user. For the aforementioned reasons, it was decided to exclude blood pressure measurement devices from the current application and consider them only as future platform expansions.

Concerning the response of the mobile application the experimental testing demonstrate the high quality of the related services of providing information and entertainment content, as well as answers to simple questions. This is due to the high speech recognition accuracy, even in the case of the users with increased level ID, i.e. mild and severe. Apart from the effective speech recognition, the smartphone application of our system provided moderate user familiarity, and high usability. According to our study, one IID could use the smartphone virtual assistant independently, while six others could use it with prompts. Our findings support the notion that IID could use computers (73.3% Google) and social media (46.7%, Facebook, 13.3% Instagram), but since more recent times and are consistent with other studies [118]. Special attention should be given to individuals who require training before using the proposed application. Additionally, alternatives should be provided for those who may not want to interact with Clio.

Our study limitations are the small number of participants but also the fact that we had not participants from a different environment or organization etc. It would be helpful to test our system also in family or other institutional environments. Obstacles for many individuals with intellectual disabilities to use our system is the ongoing assistance they need from their family members or caregivers (40% of the study participants) and the high cost of devices such as computers, tablets, smartwatches, and smartphones.

The proposed integrated system intends to help and support people with mild-to-moderate-to-severe ID to be successfully involved in leisure and creative activities, enhancing their self-confidence and self-sufficiency, promoting their healthy and well-being and enabling their inclusion in educational and socio-cultural activities. It also provides a supportive tool for their family and carers in occupational activity and support centers, but also other residential environments.

Concerning future work, accelerometer measurements might be added for implementing additional desired features like fall, or seizure detection, however, such applications require a high sampling rate that might result in draining the smartwatch battery quickly. Furthermore, as part of the final evaluation, the System Usability Scale (SUS) will be used in order to assess the usability of the proposed web-based recommendation service.