Outcome Measures Utilized to Assess the Efficacy of Tele Rehabilitation for Post‐Stroke Rehabilitation: A Scoping Review

1. Introduction

The use of innovative technology for the treatment of cognitive and motor impairments in stroke during the critical golden hour is of paramount importance [1]. Recently, the use of telerehabilitation (TR), which we define as the ability to provide assessment and intervention to people who require rehabilitative services via telecommunication, has emerged as a substitute for in-person therapy [2]. Recent studies have shown that TR can positively affect motor functions such as balance, mobility, and postural control [1,3].

TR offers a solution to address some of the accessibility challenges faced by individuals living with stroke [2,4]. A study found that TR interventions for stroke found no change between telehealth and face-to-face interventions for activities of daily living, balance, and upper extremity involvement [5]. Within TR, communication between patients and qualified rehabilitation professionals is facilitated via technologies like telephones and internet-based videoconferencing. Assessing the efficacy of these interventions is crucial to advancing the field of TR [1]. Many tools have been developed to assess both the outcomes of stroke and the efficacy of post-stroke interventions [6].

A study showed that there is a growing need for improvements in stroke care [7]. This study provided strong evidence supporting the effectiveness of both virtual reality (VR) and TR in enhancing stroke care, offering valuable guidance on selecting appropriate outcome measures for assessing the impact of these interventions on stroke survivors and their families [7]. A recent literature review recognized numerous assessment tools utilized in stroke therapy [8]. Another review of outcome measures utilized in randomized controlled trials (RCTs) identified 30 distinct measures documented in RCTs, which gauged the efficacy of interventions in stroke therapy [9]. The adequacy of TR relative to the status quo is confirmed when outcome measures demonstrate no significant decline in performance compared to traditional treatment [5]. Thus, choice of appropriate outcome metric to utilize in research and in clinical practice is imperative.

It is important to note that when selecting outcome measures for clinical observation for patient improvement, the consideration should assess not just impairments in motor function, but also encompass various factors such as the patient’s lifestyle and daily preferences [9]. There are numerous advantages to employing standardized outcome measures, which include the ability to identify patients at risk of experiencing adverse or unfavorable outcomes, determining the most effective interventions tailored to specific contexts, and assessing organizational performance [2]. Clinicians have supported the utilization of standardized tools in therapy for several years. Diana’s work in 2017 emphasized the importance of clear outcome measurements with a focus on TR and VR [10]. However, there remains a lack of consensus regarding the use of outcome measures to enable meaningful comparisons across interventions and studies [4]. This gap in consensus persists from January 2015 until the present day, especially in the context of TR. Considering the COVID-19 pandemic, during which the healthcare industry relied heavily on telerehabilitation interventions, there is a pressing need for establishing a consistent approach in this regard [11].

The current review is the first to our knowledge that seeks to elucidate the outcome measures applied in TR, a field that is rapidly evolving. With the aim of establishing comprehensive guidelines for the utilization of outcome measures in TR, particularly within the realm of stroke rehabilitation, we have conducted a scoping review that systematically synthesizes the prevalent outcome measurement practices. The present study aims to delineate the findings from this scoping review.

2. Methods

The Arksey and O’Malley framework from the University of York was used as guidance for a methodologically rigorous approach to systematically review the outcome metrics utilized to evaluate the efficacy of TR [12]. The York framework has been used broadly in knowledge synthesis trials, and consists of five stages as follows: Stage 1) Classifying the research question; Stage 2) Recognizing relevant studies based on the research question; Stage 3) trial selection; Stage 4) Charting the information within the selected trials; and Stage 5) Organizing, summarizing and reporting the findings of the scoping review. The research questions for the current review were as follows: which outcome measures are used in TR stroke therapy trials and at what time points are they controlled (admission, discharge, and follow-up of the patient) subsequent to a stroke? Which functions from the International Classification of Functioning (ICF) are assessed in the outcome measures?

2.1. Eligibility Criteria

The inclusion criteria for this scoping review consisted of trials: 1) involving stroke patients, 2) recounting a rehabilitation intervention using TR, 3) written in English, and 4) published after January 2015. The exclusion criteria were: 1) papers published in non-English languages, 2) papers not reporting outcome measures, 3) papers only reporting laboratory measures, 4) discussion and protocol papers or commentary and qualitative studies, 5) poster presentations, abstracts, or papers without enough information about the treatment, and 6) papers only reporting the change and development of the technology. The search was completed using study design or date of publication.

2.2. Search Strategy

The literature search was done by a librarian professional in the field of therapy, which included: PubMed Embase, The Cochrane Central Register of Controlled Trials (Cochrane Library), the Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Physiotherapy Evidence Database (PEDro) (until July 2023) to classify potentially related studies.

2.3. Data Collection Process

Two of the reviewers (MGN and MF) independently investigated the titles and abstracts extracted from the database searches to determine whether they fit the inclusion criteria. Disagreements about the inclusion or exclusion criteria of a specific paper based on the review of its abstract were determined by reaching a consensus or consulting a third reviewer (AS). Data extraction arrangements were established based on the current literature in the field and on the questions of the research. Extraction of the data was based on essential information according to questions of the current review such as a) the study’s authors, b) the year of publication, c) the objective(s) of the trial, d) the design of the trial, e) country, f) outcome measures reported, g) patient characteristics (e.g. age, sex, socio-economic status, level of education, motor functional level, the phase of the stroke, type of the stroke), h) related ICF domains, i) period of time at which the assessment is taken (e.g. admission, discharge, follow-up), j) technology used for TR, and k) details on the TR intervention. The outcome measures were categorized according to the ICF domains [12].

2.4. Critical Appraisal of the Included Articles

The modified Critical Appraisal Skills Programme (CASP) tool [13] was used for assessing the quality of each of the included studies by the three reviewers (MGN, MF, and AS). The CASP tool is an instrument used for evaluating the strengths and limitations of any qualitative research approach [14]. The tool has ten questions that each emphasizes different methodological domains of a qualitative study. The identification of the research questions, the relevance of the methodology (including study design), description of the population and sample size, outcomes, suitability of analysis methodologies, relevance, and clarification of results. Information was obtained from studies achieving scores greater than 50% according to the CASP scoring system.

2.5. Quality Assessment

We used the CASP tools for assessing the quality of studies, primarily case-control studies and clinical trials. The CASP RCT checklist evaluates 11 critical criteria:

1)

Did the study address a clearly focused research question?

2)

Was the assignment of participants to interventions randomized?

3)

Were all participants who entered the study accounted for at its conclusion?

4)

Was blinding appropriately addressed for participants, assessors, and therapists?

5)

Were the study groups similar at the start of the randomized controlled trial?

6)

Apart from the experimental intervention, did each study group receive the same level of care (i.e., were they treated equally)?

7)

Were the effects of intervention reported comprehensively?

8)

Was the precision of the estimate of the intervention or treatment effect reported?

9)

Did the benefits of the experimental intervention outweigh the harms and costs?

10)

Could the results be applied to your local population/in your context?

11)

Would the experimental intervention provide greater value to the people in your care than any of the existing interventions?

The CASP case-control study checklist also includes 11 questions:

1)

Did the study address a clearly focused issue?

2)

Did the authors use an appropriate method to answer their question?

3)

Were the cases recruited appropriately?

4)

Were the controls selected appropriately?

5)

Was the exposure accurately measured to minimize bias?

6)

Aside from the experimental intervention, were the groups treated equally, and did the authors account for the potential confounding factors in the design and/or in their analysis?

7)

How large was the treatment effect?

8)

How precise was the estimate of the treatment effect?

9)

Are the results credible?

10)

Can the results be applied to the local population?

11)

Do the results of this study fit with other available evidence?

Responses to these questions were recorded as ‘‘Yes", “No”, or “Can’t”. In the current review, seven studies were evaluated using the CASP RCT checklist (Table 1).

In addition, when appraising other studies using the CASP case-control study checklist, questions 4 (Were the controls selected appropriately?) and 6 (Aside from the experimental intervention, were the groups treated equally, and did the authors account for the potential confounding factors in the design and/or in their analysis?) were deemed not applicable since the reported trials were uncontrolled trials. Thus, the total number of questions for the latter studies was nine rather than 11. Sixteen out of the 23 trials had scores between 7 and 9 out of 9, with only two studies scoring 7. Six of the included trials had a score between 8 and 9 out of 11, whereas only four studies scored 7.

3. Results

The exploration of the electronic databases recognized 550 manuscripts after duplicate studies were removed. After screening of the titles and abstracts, 136 studies remained. After a full-text review process, 110 articles were excluded, leaving a total of 24 included studies. Reasons for exclusion of studies are depicted in Figure 1. This study was done in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [15].

3.1. Included Studies

The current scoping review encompasses a comprehensive analysis of 24 trials. This review is organized into three key sections: a) essential characteristics of the trials, which includes details about the authors, location, publication year, study design, subject characteristics, type of stroke, TR explanation, and the numerical score of the quality of the studies, b) TR outcome measures used in assessing post-stroke patients, and c) areas of the ICF covered by these outcome measures. The included trials were published between 2015 and 2023, and most of the trials were conducted in the USA and Canada. The most common study designs were quantitative approaches such as RCTs, CTs, case studies with one group and two groups with pre and post-test intervention (Table 1).

3.2. Participant Characteristics

The study participants primarily consisted of males (335) who had experienced various stroke conditions, including ischemic, subacute, and chronic stroke with symptoms such as hemiparesis, aphasia, and other neurological disorders. These individuals were willing and consenting to begin a rehabilitation protocol. All studies provided detailed information on age, gender distribution, and the total number of participants. Two of the studies included a single case study involving post-stroke patients (Table 1). All of the studies used TR intervention and two studies used TR with VR. The TR interventions were provided via various modalities, including video games, an internet-connected computer and laptop, TR application, serious games, and robot-based TR (Table 1)

3.3. Frequently Used Outcome Measures

A total of 20 outcomes were used in the scoping review (15 outcomes in TR studies and five outcomes in TR studies with VR). The most used outcomes were the Fugl–Meyer Assessment of the Recovery of patients with stroke (FMA), balance, and motor function in the upper limb function is mostly outcome in studies. All outcome measures were used pre- and post-protocol based on TR (Table 2).

3.4. ICF, Disability, and Health Domain

The ICF serves as a framework comprising domains or categories, offering valuable guidelines for reporting functioning, performance, and health in clinical assessments. In the current study, none of the trials employed the ICF guidelines for outcome measurement, but encompassing aspects of both upper and lower limb function, structural aspects, and physical activity. The majority of the pooled studies focus on upper limb function (trunk mobility and functional recovery) [16,17,18,19,20,21] and some studies focus on lower limb function (balance and gait) [16,22,23,24,25,26]

4. Discussion

In recent years, TR has emerged as a new technology for treating and rehabilitating stroke patients [26]. In the current review, we have identified more than 20 outcome measures that illustrate a broad range of assessments utilized in trials focused on stroke rehabilitation with interventions provided through TR. Among these measures, the most commonly used was the FMA. FMA is a performance-based deficiency index and is designed to measure motor function, balance, awareness, and joint functioning in stroke patients. It serves multiple purposes, including measuring motor recovery, assessing disease severity, and aiding in treatment planning and evaluation.

In contrast, other studies have employed various other tools to assess a common outcome such as balance [7]. These tools encompass diverse measurements, including gait speed, Barthel Index (BI), Berg Balance Scale (BBS), Stroke Impact Scale (SIS), and quality of life (QOL) metric. Importantly, the FMA has demonstrated outstanding reliability in both inter-rater and intra-rater assessments, exhibits strong construct validity, and is highly responsive to detecting changes in patient’s conditions. The BBS is a reliable tool, but it may not be sensitive enough to detect subtle yet clinically significant changes in balance in individual subjects, particularly those recovering from stroke [15]. It is a relatively inexpensive test and can be used with a wide range of populations, including healthy individuals and patients. It evaluates balance through a comprehensive assessment that encompasses two distinct dimensions, static and dynamic, via a structured questionnaire [28].

Gait analysis is another valuable measurement that was utilized in five of the included studies to meticulously assess details of step and gait speed in stroke patients [29]. In addition, the Stroke Impact Scale (SIS) is a widely used measure due to its reliability, validity, and sensitivity to change [30]. The SIS contains a question to evaluate the patient’s global perception of their percentage of recovery [30]. Another frequently utilized measure that was used in studies is the Barthel Index (BI). However, there is a strong need for greater consistency in methods, content, and scoring across studies, given that the “BI” acronym is associated with various assessment methodologies. For example, some studies have adopted a 10-item scale, scoring on a range of 0 to 100 with 5-point increments [31]. This approach has been used in several multicenter stroke trials, and we call for more uniform application of this tool for stroke trials. Consistency in result reporting will allow for more appropriate pooling of data for literature review and meta-analysis.

In general, all the aforementioned outcome measures aim to capture important changes in patients who are undergoing stroke rehabilitation, whether by TR or more traditional means. Importantly, most studies have highlighted that patient satisfaction plays a pivotal role in their recovery and motivation to continue with rehabilitation to regain function. Surprisingly, only two studies incorporated a thorough assessment of patient satisfaction and motivation, using tools like the Client Satisfaction Scale (CSS) and the Canadian Occupational Performance Measure (COPM). Upon examining the satisfaction levels of patients who underwent TR following a stroke, the results unequivocally indicate that TR can be a highly effective intervention in the realm of rehabilitation.

Our scoping review identified various evaluation questions that pertained to changes in health service utilization, intervention costs, and the utilization of comprehensive assessment tools to gauge aspects of patient safety, comfort, ease of use, and the efficiency-related consequences resulting from interactions with the technology [31]. This scoping review focused on motor functions such as upper-extremity function, balance, and postural control, yielding outcomes similar to those observed in previous research, such as the study conducted in 2017 [7]. Notably, the trial of Tate et al. found a limited number of studies (8.8%) that assessed specific motor, sensory, and other bodily functions [32]. It is worth mentioning that the majority of the studies in the Tate et al. publication predominantly evaluated domains related to mental function [32]. In contrast, our scoping review identified only two studies that used the Mini-Mental State Examination (MMSE). Future studies should prioritize outcome measures that support ICF domains using TR. Adhering to the Canadian Best Practice Recommendations for Stroke Care can comprehensively cover the various aspects of the ICF framework during both the short- and long-term recovery in stroke patients.

5. Conclusions

Our review included RCTs that provided essential information regarding participant demographics, including age and sex, as well as details about the interventions and the specific type of TR employed for rehabilitation. A majority of these studies assessed outcomes related to motor function, consistently reporting improvements in this domain. However, it is important to note that most studies did not include information about the cost implications of the interventions, which could provide valuable insights for healthcare providers, clinicians, patients, and their families when making decisions based on using new technology with TR. Future studies should emphasize measuring the utilization and feasibility of these outcomes within the context of TR while also providing detailed cost-related information. Furthermore, future studies should investigate the standards that guide the selection of outcomes by clinicians and investigators. It is crucial to explore the reasons behind the exclusion of certain outcomes, such as the need to establish new protocols for professionals, ensuring the availability of assessment tools in the same language as the patients, managing the time required for assessments, and addressing equipment-related prerequisites for the utilization of specific tools. Understanding and addressing these factors will contribute to the improvement of outcome selection processes in TR and related research.