German eHealth Impact Questionnaire for Online Health In- formation Users with Multiple Sclerosis: Instrument Valida- tion Study

Introduction: This study aimed to validate the psychometric properties of the German eHealth Impact Questionnaire (eHIQ-G), which is divided into two independently administered and scored parts. Methods: 162 people with MS browsed one of two possible websites containing information on MS and completed an online survey. Internal consistency was assessed by Cronbach’s alpha and structural validity by Confirmatory Factor Analysis. Construct validity was examined by assessing correlations with the eHealth Literacy Questionnaire and the General SelfEfficacy Scale. Moreover, the mean difference of the eHIQ-G score between the two websites was investigated. Results: Cronbach’s alpha for the eHIQ-G subscales ranged from .833 to .885. The eHIQ-G part 1 achieved acceptable levels of goodness-of-fit indices, whereas the fit for the eHIQ-G part 2 was poor and likewise for the alternative modified models. The correlations with the reference instruments were 0.08 – 0.62 and as expected. Older age was related with lower eHIQ-G part 1 score, whereas no significant effect was found for education. Although not significant, the website ‘AMSEL’ reached higher mean scores on eHIQ part 2. Conclusion: The eHIQ-G has good internal consistency, and sufficient structural and construct validity. It will facilitate the measurement of the impact of websites providing health information.


Introduction
Among chronically ill people, persons with multiple sclerosis (MS) are among the most frequent internet users. Thus, they are confronted by an overwhelming amount of online health information differing in quality and aim [1][2][3]. Methods to evaluate electronic Health (eHealth) tools, such as mobile apps, online-portals for patients, and other internet-based software or programs used to help patients monitor and manage their health have recently started to emerge [4]. A framework has been proposed by Allison et al. (2019) suggesting to evaluate website attributes, such as usability, content, web design, and functionality [5]. Currently, there is limited but increasing research looking at the potential impact of websites presenting individual patient experiences (PEx) [6][7][8][9][10][11][12][13][14][15][16], since PEx are increasingly exchanged between patients and more frequently incorporated into health information websites [17]. It is crucial for developers and providers of online health information, especially experiential information, to understand its potential impact on knowledge, feeling of being supported, preparation for health decisions and behavioral outcomes on the users [18]. Therefore, instruments to measure these dimensions are urgently needed.
In 2013, the eHealth Impact Questionnaire (eHIQ) was developed to measure selfreported impact of eHealth tools on the user [18] and validated as an English version in 2015 [19]. To our knowledge, only a validated Dutch and Hebrew version of the eHIQ exists [20,21]. The eHIQ measures patient's general attitudes towards using the Internet to obtain health information. Furthermore, it assesses impacts on patients after viewing a specific website containing different types of material (e.g., patients' experiences, and scientific facts and figures). The impacts refer to 1) the extent to which patients gain confidence in discussing their health with others and the ability to identify with others; 2) the perceived ease of use, ease of understanding, trustworthiness and appropriateness of website content; and 3) the extent to which the patients better understand their health condition, feel reassured, and motivated to take action [7,18,19]. The eHIQ has already been used in several studies to evaluate eHealth tools, such as an online patient platform for communicating laboratory test results [22], a website to support families of burn-injured children [23], and a website providing narratives on prostate, breast and colorectal cancer [11]. The portal for laboratory test results achieved a high score on the usability scale of the eHIQ, but the portal could help the patients only slightly to take action in managing their own health. The website providing burn-specific information was rated very positive and slightly better than the former portal. The information on the website providing narratives was considered valuable and trustworthy by the majority of participants [11,22,23].
This study aimed to validate the eHIQ in a German population of eHealth users with MS by using confirmatory factor analysis and comparing the ratings for two websites of different complexity.

Materials and Methods
The Materials and Methods should be described with sufficient details to allow others to replicate and build on the published results. Please note that the publication of your manuscript implicates that you must make all materials, data, computer code, and protocols associated with the publication available to readers. Please disclose at the submission stage any restrictions on the availability of materials or information. New methods and protocols should be described in detail while well-established methods can be briefly described and appropriately cited.
Research manuscripts reporting large datasets that are deposited in a publicly available database should specify where the data have been deposited and provide the relevant accession numbers. If the accession numbers have not yet been obtained at the time of submission, please state that they will be provided during review. They must be provided prior to publication.

Recruitment and Procedure
As we intended to use the scale in a clinical setting [24], participants were Germanspeaking persons with MS or people with suspected MS who were aged ≥ 18 years, and who had access to the Internet. The necessary sample size for validating questionnaires is contentious [25]. In accordance with our resources and the COSMIN Guidelines (Consensus-based Standards for the selection of health status Measurement Instruments) [26], we aimed to reach a sample size of 150.
Open recruitment took place from November 2019 to Mai 2020 through newsletters of the MS day hospital at the University Medical Center Hamburg-Eppendorf and as part of regular newsletters of four regional associations from the German Multiple Sclerosis Society (DMSG): Hamburg, Baden-Württemberg, Schleswig-Holstein, and Lower Saxony.
Persons with MS were invited to access an anonymous online survey by clicking on an electronic link. After reading the patient information and giving informed consent, patients were asked to fill out the following measures: the eHealth Literacy Questionnaire (eHLQ), eHIQ-part 1 and the General Self-Efficacy Scale (GSE). Afterwards, participants were directed to spend at least 10 minutes browsing either the section 'living with MS' of the website of the DMSG Baden-Württemberg, called AMSEL [27] or the whole website of the DMSG Hamburg [28]. In addition to factual information, the AMSEL also contained explanatory films about living with MS from patients and health professionals. The DMSG Hamburg website contained only factual information at the time of the study. The websites were chosen to test the items with health information based on facts, figures and personal experiences. After browsing one of the websites, patients had to answer the eHIQ-part 2 and demographic as well as MS-related questions.

Measures
The eHIQ is divided into two parts. The 11-item eHIQ-part 1 has to be completed before accessing the website to be evaluated. It consists of two subscales 1) attitudes towards online health information and 2) attitudes towards sharing health experiences online. The 26-item eHIQ-part 2 measures the impact of using a specific health-related website on three subscales: 1) confidence and identification, 2) information and presentation and 3) understanding and motivation. Response options range from 1 ('strongly disagree') to 5 ('strongly agree'). The eHIQ part 2 must be administered after accessing the website to be evaluated. However, both parts could also be used in succession after assessing the website as implemented by another study using the eHIQ [11]. The scores were converted to a 0-100 metric. The total eHIQ score for part 1 and 2 was calculated as the sum of all subscale scores, divided by the number of subscales. Higher scores correspond with more positive responses [18,19]. The translation of the eHIQ into German was performed in the context of a medical dissertation [29]. The procedure consisted of the steps: translation, review, adjudication, pre-test, and backwards-translation. The eHIQ-G was pretested in a convenience sample of 25 participants.
The eHLQ is a validated measure of eHealth literacy in English and Danish language covering user interaction with a given eHealth system and the user's experience of engaging with it. The eHLQ is valuable for the characterization and understanding of digital health literacy in a broad range of target groups. It contains 35 items in seven domains: 1) using technology to process health information, 2) understanding of health concepts and language, 3) ability to actively engage with digital services, 4) feel safe and in control, 5) motivated to engage with digital services, 6) access to digital services that work, and 7) digital services that suit individual needs. Response options for all items range from 1 (strongly disagree) to 4 (strongly agree). The subscale scores are calculated summing up the scores of each item and dividing it by the number of items [30]. We used the eHLQ from the German eHLQ-validation study after back-and forward translation and a qualitative pre-test.
The 10-item GSE scale was developed and validated to assess a general sense of perceived self-efficacy. Responses are made on a 4-point scale from 1 (not at all true) to 4 (exactly true). The total score is calculated by summing up all item scores. The total score ranges from 10 to 40, with a higher score indicating more self-efficacy [31].
Demographic data such as sex, age, educational level and highest professional qualification were collected as well as MS-related information, e.g. the disease course, years since diagnosis and the 9-item-'Patient Determined Disease Steps' (PDDS), which asks for the patient-reported walking ability and disability (from 0 = normal to 8 = bedridden) [32].

Data analyses
The analysis was performed in SPSS (version 25.0; IBM Corp.) and SPSS Amos (version 26.0; IBM Corp.) software. All analyses were carried out on complete cases. For sample description, continuous variables are described using mean and standard deviation (SD) and categorical items are presented as counts and percentages. To examine the internal consistency reliability of the five subscales Cronbach's alpha (α) was estimated. A Cronbach's alpha value of > 0.7 was considered adequate [26].
Confirmatory factor analysis (CFA) was applied to investigate construct validity. The Full Information Maximum Likelihood estimation was used to estimate model parameters and to examine goodness-of-fit of all the CFA models with: the Root [33,34]. An exploratory factor analysis (EFA) using Oblimin rotation and principal component analysis was run to investigate an alternative to the original structure.
Moreover, convergent validity was assessed by testing hypotheses about expected relationships with eHLQ and GSE by calculating Pearson correlation coefficients (r). Correlations with instruments measuring related, but dissimilar constructs (eHLQ, GSE) should be 0.30-0.50 [26]. Convergent validity was considered adequate if at least 75% of the correlations were as expected. P values higher than 5% are interpreted as statistically significant. Hypothesis 1: Particular subscales of the eHIQ-G correlate with subscales of the eHLQ.

Sample characteristics
163 persons with MS were enrolled in the study ( Table 1). The response rate (59.3 %) was calculated as the number of returned questionnaires (N=163) divided by the total sample who logged in the survey (N=275) (Figure 1).

Figure 1. Participant flow chart
Most of the participants were female (71.0%), with a mean age of 51 years. The level of education was high with 65.4% reporting a high school degree. 58.0% were having a vocational degree as highest professional qualification. Nearly all participants had a definite MS diagnosis, while 4.9% were suspected having MS. 54% of those with definite MS had a relapsing-remitting MS course. The patients had lived with the disease on average for 14 years and had on average a PDDS of 3.1 (moderate to gait disability). Most participants (65.4%) decided to spend time on the website of the DMSG Hamburg, while 34.6% browsed the website of the AMSEL. Regarding the use of those websites before participation in this study, 77.4% had already used the DMSG Hamburg website, while 80.4% had already used the AMSEL website.

Internal reliability
All subscales showed good internal consistency with Cronbach's α from .833 to .885 (Table 2). Overall internal consistency for the entire eHIQ was .926.

Structural validity
The CFA was run on 149 complete cases that were only participants for whom we had no missing data on the eHIQ scale ( Figure 1). The analysis was performed on the five factors and 37 items of the eHIQ-G scale ( Figure 2). All items had a standardized factor load (β) of ≥ .50 and a critical ratio value (C.R.) of ≥ 1.96 (p <.05) indicating a good model identification except for eHIQ 2 item 10 and 25 ( Table 2). Results of the CFA on 149 complete cases for the whole eHIQ-G including part 1 and 2 suggested poor fit to the data (RMSEA=0.09, SRMR=0.09, CFI=0.75, TLI=0.73). Solely CMIN/DF=2.26 showed an acceptable fit.
Additionally, two CFAs were run separately for each part of the eHIQ. The goodnessof-fit indices of the eHIQ 1 model were as follows: RMSEA=0.12, SRMR=0.07, CFI=0.91, TLI=0.88, CMIN/DF=2.63. All the fit indices ranged from satisfactory to poor. An EFA was run to investigate an alternative to the original structure of the eHIQ 1. The Kaiser-Meyer-Olkin (KMO) measure verified the sampling adequacy for the analysis, KMO= 0.87. The EFA resulted in the same two factors with the same items (predictors) as the original model.
The fit for the eHIQ 2 model was poor (RMSEA=0.12, SRMR=0.10, CFI=0.74, TLI=0.71, CMIN/DF=3.00). Inspecting the modification index output (Appendix A), which showed covariances that could be incorporated into a re-specified model to obtain superior goodness-of-fit, covariances were found between items 14 and 15 as well as between items 20 and 23. After removing items 15, 20 and incorporating item 10 into the subscale eHIQ 2.3 the fit for the alternative model (Appendix B) was better, but still not acceptable (RMSEA=0.12, SRMR=0.10, CFI=0.77, TLI=0.75, CMIN/DF=2.75). The EFA for the eHIQ 2 suggested five factors, which were not clearly interpretable and had many items that had double loadings. Furthermore, we applied the alternative Dutch 3-factor structure [20] (Appendix C) in our CFA and resulted in better, but also unacceptable fit indices (RMSEA=0.11, SRMR=0.10, CFI=0.76, TLI=0.74, CMIN/DF=2.81). The path diagrams (Figure 2, Appendix B and C) showed partly high correlations between the subscales (latent factors). As this was already found by the English validation study [19], we have decided to stay with the original 3-factor model of the English eHIQ 2.

Convergent and discriminant validity
Descriptive data for the GSE, eHLQ subscales and eHIQ part 1 are shown in Table 3. Participants considered their knowledge and skills related to eHealth literacy to be moderate. The mean score for participants' general attitudes towards using the Internet to access health information and their ease with using online (experiential) information for learning and gaining support (eHIQ part 1) was medium. Despite the concentration of eHIQ scores at the positive end of the construct, the distributions were sufficiently symmetric. The eHLQ subscales (with data of the subscale 1 concentrating on the positive end, and eHLQ subscale 4 on the negative end) as well as the GSE were approximately normally distributed.  Figure 3 illustrates the eHIQ part 2 subscale scores and the sum index score of each website. Data were displayed in the boxplot by the minimum, the maximum, the median, the mean, the first and third quartiles and the outliers (dots). Both websites gained moderate sum index scores on the eHIQ part 2. The websites achieved the highest score in subscale 2.2, which reflects users' trust and suitability of the website content. Participants' confidence to discuss their health condition with others after browsing the websites and their ability to identify with the websites (eHIQ 2.1) was moderate. The score on subscale 2.3, which reflects the understanding and learning about relevant health information and motivation to act was moderate as well. The t test indicated no significant difference in mean in eHIQ part 2 sum index score between the websites of DMSG Hamburg and AM-SEL (t 149= -0.45, p=.35, mean difference=-0.40, 95% CI=-6.08 -4.04). No significant differences in means of the subscale 2.1 (t 154= -1.00, p=.32, mean difference = -3.01, 95% CI= -9.08 -3.00), subscale 2.2 (t 156 = 0.95, p=.35, mean difference=2.35, 95% CI= -2.56 -7.25) and subscale 2.3 (t 129.55 = -1.08, p=.28, mean difference= -3.05, 95% CI= -8.96 -2.87) between DMSG Hamburg and AMSEL could be proven. However, descriptively the average mean eHIQ part 2 sum index score as well as in subscales 2.1 and 2.3 for the AMSEL website were descriptively higher than the average sum index score for DMSG Hamburg. Moreover, relationships (Pearson's correlation coefficients) between eHIQ subscales and the selected reference measures were examined to assess construct validity. Results confirmed our expectations that almost all scales are significantly related: a large positive correlation was found between the eHIQ 1.1) and eHLQ 1) and 5) as well as a positive moderate to small correlation between the eHIQ 1.2) and eHLQ 3), 6) and 7). Correlations between eHIQ 2.1) and eHLQ 5) and 7) were small. There were either small (r<0. 20) or no significant (p=.17) correlations between eHIQ 2.2) and eHLQ 2) and 4). Only a positive small correlation was found between eHIQ 2.3) and eHLQ 5). The eHIQ 2.3) did not correlate significantly (p=.32) with the GSE score (Table 4).

Discussion
Persons with MS, like other people with especially chronic conditions, increasingly search for health information on the Internet [35]. Among chronically ill people, persons with MS are among the most frequent internet users being confronted by an overwhelming amount of web-based information of variable quality and aims [1,2]. As they are young, now mostly digital natives, mostly not substantially impaired in the early phase of the disease [36,37], persons with MS represent a prototypic, ideal eHealth population. Web-based health information and interventions are a substantially emerging area of providing treatment for various conditions or supporting self-management [38][39][40] which has been even accelerated in the COVID-19 pandemic [41]. However, critical appraisal of eHealth information is a challenge and the real impact on health behavior and health state is a matter of emerging research. Besides ensuring the quality of content and web design, health website developers and health care providers should also consider the impact that their websites might have on the users [12,[18][19][20]. As a valid and reliable instrument was lacking in Germany, this study aimed to examine the structural validity, internal consistency, and construct validity of the eHIQ-G. The results show that the German eHIQ has sufficient structural validity, internal consistency, and construct validity as well as conditionally sufficient structural validity.
The CFA analyses showed satisfactory fit indices for the original eHIQ-G part 1. Similar results have been found for the Dutch version of the original model. The fit for the eHIQ-G part 2 was poor and likewise for the alternative 3-factor model. Neijenhuijs et al. reported bad fit indices for the original model of eHIQ part 2, too. An alternative 3-factor structure was investigated and resulted in a good model fit [20]. No model-fit-indices were reported for the Hebrew version [21]. We have investigated an alternative model based on the modification index output as well as on the alternative Dutch factor structure which resulted in better, but still unacceptable fit indices. The developers of the eHIQ found that 12 items of the eHIQ part 2 were loading on more than one subscale. However, items were allocated to the subscale on which they loaded most highly and made conceptually sense [19]. To have comparable results with other countries and to avoid a shift in the model's meaning from a theoretical standpoint, we decided to stick to the original factor structure of eHIQ part 2.
Evidence for sufficient internal consistency of the eHIQ subscales was indicated by a Cronbach's alpha of >.70 [26]. The internal consistency found in this study was comparable to the internal consistency found in previous studies [19][20][21].
As no validated reference inventory in German language exists, which measures the self-reported impact of eHealth tools on the user, we selected related, but dissimilar construct measurement tools such as the eHLQ and the GSE. Predominantly all expectations towards hypotheses one to five were confirmed. However, many of the correlations were small. Our results are difficult to compare with the results of the other validation studies [19,20], since others did use other reference measures. The authors report finding small to acceptable correlations for convergent and divergent validity. We were not able to use these questionnaires as they were neither available in German nor validated. We could only partially confirm discriminant validity (hypotheses 3-4): older age was related with lower eHIQ part 1 score, whereas no significant effect was found for education on eHIQ part 1. Kelly et al. (2015) found no significant difference for age [19]. No significant difference was found for the browsed websites among all subscale scores of the eHIQ part 2 (hypothesis 4). This did not match our assumption that the AMSEL being richer in different types of information, e.g., providing MS patients with videos on how to manage their daily lives, would get higher scores on the average mean eHIQ part 2 sum index score. This study has several limitations. As with most web-based studies, a selection bias related to recruitment of volunteer participants is also present in our study. Second, although the sample size was adequate according to KMO other sources [42] say that more data are needed to perform CFA. In this study, only persons with MS were chosen to represent the target population of the eHIQ-G, which are patients with different kinds of health conditions. Other patient groups should be included to test whether the eHIQ-G is applicable to a range of different conditions. Besides, we could show discriminant validity only based on differences of the eHIQ scale performance at different ages. We did not assess discriminant validity with reference inventories to be regarded as measuring distinct constructs [43], as we were lacking scales measuring related constructs based on theory or prior empirical observations. Finally, we used an anonymous survey for pragmatic and data protection reasons while collecting data. Therefore, we did not test for test-retest reliability to examine the consistency over time.

Conclusions
The eHIQ-G is a reliable and valid inventory with acceptable psychometric properties assessed in a group of patients with MS. Lastly, this is the first study evaluating the German version of eHIQ in MS patients and suggesting a 37-item model, which is possibly able to describe different important aspects of health websites with various styles of information from a user perspective. The eHIQ-G is proposed for usage in future studies on the impact of websites containing various styles of health information. The eHIQ-G can be used to reflect on user's general attitudes towards health-related websites (eHIQ part 1) and on the website's design, credibility, reputation, and the possible impacts (eHIQ part 2). Therefore, it can be used to test and improve websites with health information. Additionally, the inventory is promoting research and comparison among different websites, since it has already been translated in several languages and validated in several groups. Therefore, we recommend considering the eHIQ in future studies assessing the impact of eHealth tools on its users. Funding: This research was funded by Roche Pharma AG, grant number: 29623. The sponsor was kept apprised of the process but had no role in the study design or conduction to uphold the scientific integrity and quality of the study.