Co-Design/Co-Development and Evaluation of an Online Frailty Check Application for Older Adults: Participatory Action Research with Older Adults

1. Introduction

Frailty is a complex age-related clinical condition characterized by a decline in physiological capacity across several organ systems [1,2]. In Japan, to help community-dwelling older adults monitor their own frailty status, we developed a frailty check (FC) program based on evidence from Kashiwa cohort study, which was a large-scale cohort study of older adults in Kashiwa City who were not certified as requiring long-term care [3]. This FC program consists of self-assessments for sarcopenia (Yubi-wakka test) [4], an 11-item self-reporting questionnaire, and physical measurements of skeletal muscle mass, calf circumference, and handgrip strength. Between April 2015 and February 2020, a total of 8,855 community-dwelling older adults in 47 local government prefectures participated in the FC program. Notably, this citizen-centered program is staffed by FC supporters, who are community-dwelling older volunteers. FC supporters encourage participants to improve their lifestyle and practice frailty prevention [5].

Recently, the coronavirus disease 2019 (COVID-19) pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) resulted in restrictions on going outdoors, which caused a decrease in physical activity among older adults. This decrease in physical activity can lead to sarcopenia and frailty [6,7,8,9]. Our recent study found, among Japanese community dwelling older adults, a significant decrease in the trunk muscle mass (one of the risk factors for postural instability and falls) immediately after the pandemic’s first wave (April−May 2020) [10]. Frailty caused by the COVID-19 pandemic became known as corona-frailty. Thus, to continuously monitor frailty status during the COVID-19 pandemic, a mobile application that can be used for remote check-ups is urgently required.

In this study, we aimed to co-design/co-develop and evaluate an online FC application based on a pre-existing onsite FC program with the help of FC supporters through participatory action research.

2. Materials and Methods

2.1. Older Adults: Frailty Check Supporters and Participants

In this study, 32 FC supporters were involved in the development and implementation of the online FC application. The FC supporters were community-dwelling older adults who had facilitated the onsite FC program in Bunkyo-ku and Nishitokyo-shi in Tokyo, Japan. Additionally, we recruited 20 community-dwelling older adult participants who had participated in the onsite FC program in Nishitokyo-shi, Tokyo, Japan.

2.2. Study Design

A flow diagram of the development and implementation process is shown in Figure 1. To co-design and refine the application, focus group interviews and mock tests were conducted with FC supporters. Self-efficacy questionnaires were assessed before and after the mock test. After online FC implementation, we conducted surveys on the reliability, communication, and usability of the user interface (Figure 1a).

2.2.1. Online Frailty Check Application: Device and Contents

The online FC application was conducted using a tablet device through video conferencing in a room containing a maximum of six people (Figure 1b). The application utilized the same system used by online cram schools, which enables communication between teachers and students and allows for document sharing by teachers. In the application, FC supporters acted as the teachers and projected questionnaires onto the right upper side wall of the online room. Participants would input data or answer questions using an electronic pencil while following the FC supporters’ instructions.

The program consisted of three sections: basic information (age, sex, body weight, height), Yubi-wakka test (sarcopenia test with video description [4]) and an 11-item FC questionnaire (yes/no, validated questionnaires including questions on nutrition, oral and physical function, and social activity) (Figure 1b). This structure was also used in the onsite FC program. Each section of the questionnaire is addressed in Table S1 and a video of the online FC implementation is also supplied as Video S1 (in Japanese).

2.2.2. Development and Refinement of Application; Participatory Action Research Method

To collect information from various sources, we used the participatory survey method to conduct the research. To facilitate smooth execution of the experiment and the collection of authentic and reliable data, the process was divided into two main parts. To collect primary data, we began by conducting online FC briefing sessions and conducted several user-experience-sharing meetings. The primary data was extracted from focus group interviews, discussions, and observations as outlined in Figure 1. User experience meetings for the prototype application were held over 2 days, and a total of 32 FC supporters participated. Subsequently, we made improvements based on the opinions and feedback received.

Data analysis was conducted using the following steps. First, we coded the interview data using first-order codes: (1) reliability, (2) social interaction, and (3) user-friendliness. Second, we added the frequency to indicate the importance of each code. After the coding and categorization, we reviewed and refined each opinion through discussion.

2.3. Questionnaires

For FC supporters, we assessed self-efficacy using a validated 16-item questionnaire on the general self-efficacy scale (GSES) consisting of three categories: positive behavior, non-anxiety about failure, and social positioning of ability [11]. For both the FC supporters and participants, we evaluated the user interface of the online FC application using relevant questionnaires. To assess the online-onsite reliability (3 items) and interaction quality (4 items), we utilized modified questionnaires on telehealth usability (Table S2) [12]. We also scored the system usability scale (SUS) [13] using 10 questionaries, with possible values ranging from 0 to 100, to examine the overall usability.

2.4. Analysis

The differences between FC supporters and participants were examined using Student’s t-test or the Mann−Whitney U-test after confirming normal distribution using the Shapiro–Wilk test. We examined the effect on self-efficacy before and after the mock test using a Wilcoxon rank-sum test, after confirmation of normal distribution by the Shapiro–Wilk test. Cronbach’s alpha was calculated to appraise the scale’s internal consistency. To analyze the relationship between the SUS and related independent variables, as well as between online and onsite FC results, we applied Pearson’s or Spearman’s correlation analysis. Multiple regression analysis was performed using SUS as the dependent variable and age, sex, education, ICT proficiency, reliability, and interaction quality as independent variables. Data were analyzed using IBM SPSS Statistics version 26 (IBM Japan, Tokyo, Japan). Statistical significance was set at P<.05.

3. Results

3.1. Frailty Check Supporters and Participants

The basic characteristics of the FC supporters and participants are shown in Table 1. Of the 12 FC supporters, seven were men. Compared to FC supporters, the participants were significantly older and had lower education levels, while subjective health and well-being did not differ between the two groups. The subjective proficiency of ICT tended to be low for the participants (P=.082). Among participants, two major reasons were identified for their participation in the online FC: (1) interest in the online FC program (40%) and (2) the need for continuous frailty checks (40%) owing to cessation of the onsite FC program during the COVID-19 pandemic (data not shown).

3.2. Development and Refinement of the Application

Opinions collected from the focus group interviews and the refinement process are shown in Table 2. Fifty-five detailed opinions were obtained and classified into five categories based on key words such as reliability, social interaction, and user-friendliness. The largest number of opinions (n=18) were related to reliability. Refinement was performed based on the categorized opinions. For example, to improve familiarity with the online questionnaire format, the onsite format of the 11-item self-reported questionnaire was adapted and modified to display one question at a time on a single page. In addition, a confirmation page was added so each participant could reconfirm all the responses. Furthermore, FC supporters expressed difficulties in online communication, such as identifying the speaker and capturing the other side’s reaction. To enhance and encourage communication, we implemented several measures, including spotlighting the person speaking, providing an explanation of the system, and removed the daily topic section before the check.

FC supporters expressed difficulty in conducting the Yubi-wakka test over the screen. To address this challenge, we created a video explaining the test together with the supporters and embedded it into the application to provide an accurate and consistent explanation similar to the onsite FC.

Furthermore, several opinions on user-friendliness were collected and improvements based on this feedback were implemented: 1) using a stylus, 2) enlargement of font size, and 3) introducing a button pressing simulator.

3.3. Self-efficacy of Frailty Check Supporters and Usability Evaluation

In the co-design and co-development of the online FC application, we investigated whether the engagement of FC supporters in these processes affected their self-efficacy. As shown in Table 3, a slight but not significant increase in the total GSES score was observed after the mock test (before: 12.0 vs after: 13.0). However, among the three GSES categories, we noticed a trend of improvement in the non-anxiety of failure category, as the proportion who answered “no” regarding (Q5) “I am more concerned about small failures than others all the time”, tended to increase after the mock test (before: 63.0% vs. after: 77.8%, P=.094). Additionally, in the social positioning of ability category, the proportion of “yes” responses to (Q3) “there are areas where I have better knowledge than friends,” significantly increased (P=.031) in comparison to results from before the mock test. Similarly, the proportion of “yes” responses to (Q1) “I have better ability than friends,” also increased (P=.063).

The overall system usability of the online FC application was evaluated using SUS, a widely used questionnaire with an acceptability and adjective range scale (Figure 2). We found that the average SUS scores both of FC supporters and participants were similar and in the “marginally high” section of the acceptability range. Additionally, the scores were above the average (68 points [15]) of the adjective ranges, which implies that the developed FC application is suitable for older adults.

Next, we examined the overall SUS score and the factors that affect the user interface. As shown in Table 4, we found that age, education level, subjective ICT proficiency, reliability, and interaction quality were significantly associated with SUS (P<.05). Furthermore, when we assessed the multiple (linear) regression analyses, we observed a significant correlation between the SUS score and the reliability of the online FC application after adjusting for covariates including age, sex, education duration, and ICT proficiency (Table 5). On the other hand, the interaction quality did not display a significant association with the SUS score.

We further validated the results of the online FC program by comparing them against those of the onsite FC, which were conducted within 4 months of each other. Pearson’s correlation coefficient revealed a significant association between the onsite and online results (P=.001) (Figure 3).

4. Discussion

In this study, we developed an online FC application for community-dwelling older adults delivered by video conferencing, during the COVID-19 pandemic. To enhance its affinity for older adults, we co-designed and co-developed it alongside FC supporters, who are older community-resident volunteers from a pre-existing onsite FC program. Our focus was on improving onsite-online reliability and online communication, two major determinants of the user interface.

To the best of our knowledge, this is the first online application for frailty checks designed by older adults, for older adults. To achieve our goal, we utilized a participatory action research design [15,16,17], consisting of focus group interviews and a mock test. FC supporters (users who acted as session facilitators) from two research fields were involved in the development of the online FC application, which achieved high scores on the SUS for both participants and FC supporters. Based on this, it is conceivable that a participatory design might be a useful methodology when developing digital healthcare interventions for older adults.

Video conferencing has been proposed as a medium for digital health intervention that can be used for communication, providing care and support, and enhancing health status, such as home-based tele-exercise for patients with chronic diseases [18,19]. In this study, we also explored the use of video conferencing through tablets and found it to be a potentially useful online tool for older adults, as they can communicate and collaborate to solve problems together during the frailty check. However, the efficiency and cost-effectiveness of video-conferencing technology should be examined in future research.

Intriguingly, we found that the FC supporters were able to strengthen their own social positioning of ability and relieve anxiety about failure through participating in the development of the application. This highlights the beneficial effects of co-design and co-development on self-efficacy for FC supporters. Furthermore, the empowerment of the FC supporters equips them with the confidence to operate the online FC platform, which subsequently could enhance the usability for all participants.

The results of this study showed that online-onsite reliability was significantly associated with SUS scores, even after adjustment for several variables including sex, age, educational level, and subjective ICT proficiency. This finding suggests that the similarity between the contents and operation of the online FC program and the pre-existing onsite FC program contributes to the safety, satisfaction, and usability for older participants. However, interaction quality was not significantly associated with SUS scores, indicating that several issues need to be addressed in online communication among older adults. For example, audio-video quality should be improved to help prevent frailty, particularly in terms of social isolation and loneliness. Moreover, enhancing online communication should be done with an understanding of the intention to use medical applications among older adults [20].

Finally, we validated the results of online FC assessment by comparing them with those of onsite FC. The significant similarity observed between the two could be attributed to the reliability and simplicity of the application (11 yes/no questions). With this observation, it is feasible that the online FC application could be used as a complementary tool for onsite FC among older participants, especially in emergency situations like the COVID-19 pandemic, or for sustained monitoring of frailty status. Although older adults cannot accurately assess their frailty status using video conferencing alone, this online FC application, which was implemented by older adults themselves, is a significant step toward the necessary training and system changes needed for adequate assessment of frailty status.

4.1. Limitations

Our study has several limitations. The primary focus was on developing the application and conducting a preliminary evaluation of its feasibility and acceptability. The evaluation study was not designed to rigorously test the efficacy of the online FC assessment in improving frailty status. Additionally, the use of non-systematic recruitment methods may have introduced bias toward participants who were more open to this type of intervention.

Despite these limitations, our study has several important implications: (1) individual FC supporters and participants may benefit from using the online FC application; (2) a participatory design approach was a useful methodology for developing a relevant, useful, and accessible tool for older adult users.

5. Conclusions

We co-designed and co-developed an online FC application using a participatory action research approach to create a reliable tool for older adults to assess their frailty. Our study found that online-onsite reliability is an important determinant affecting the usability and satisfaction of the online FC application. Furthermore, it is conceivable that enhanced empowerment of FS supporters may lead to high usability scores in participants, a win-win situation. Further investigations are needed to rigorously test the efficacy of the developed online FC application with larger sample sizes.