Leveraging Artificial Intelligence to Predict Health Belief Model and COVID-19 Vaccine Uptake Using Survey Text from US Nurses

1. Introduction

Healthcare providers worldwide became the frontline workers in battling COVID-19 by treating infected patients. Healthcare providers encountered numerous challenges increasing their risk of infection, such as inadequate Personal Protective Equipment (PPE), high workloads, and extended shifts [1]. In December 2020, US healthcare personnel began to be offered mRNA vaccines (Pfizer-BioNTech and Moderna) under emergency use authorizations (EUA) [2], a critical step toward protecting healthcare providers [3]. Several surveys conducted before the EUA documented that 8-18% of healthcare personnel expressed hesitancy about the safety and efficacy of the new vaccines [4]. Vaccine hesitancy dropped after the EUA and the vaccine rollout to healthcare personnel (HCP), yet a small minority of nurses remained vaccine hesitant even into spring 2021 [5–7]. Determining why hesitancy lingered is a challenge. Heyerdahl et al. coined ‘unspoken vaccine hesitancy’ in describing when nurses may be uncomfortable expressing their vaccine-related concerns and beliefs due to the social and institutional pressure to get vaccinated [8].

Uncovering and understanding the factors behind vaccine hesitancy is crucial to increasing vaccine confidence in healthcare institutions and mitigating the spread of COVID-19. Previous studies have shown that open-ended questions in healthcare surveys are valuable resources to elicit respondents’ concerns [9] and can reduce response biases stemming from respondent beliefs about desired outcomes [10]. Therefore, to examine what underlies vaccine hesitancy among nurses and what factors influenced uptake, as expressed in their own words, we leveraged a national longitudinal survey of US nurses and applied Machine Learning (ML) methods to the open-text comments regarding vaccines in January 2021 (winter 2021 survey) as predictors of survey respondents’ vaccination status in March-April 2021 (spring 2021 survey). Among a cohort of initially vaccine hesitant nurses who included comments, 40% received a COVID-19 vaccine by spring 2021, but 60% remained unvaccinated. To help understand why, we employed constructs from the Health Belief Model to help interpret the nurses’ rationales in their responses [11].

1.1. Gaining insight from open-field comments

Qualitative research can help understand the concerns and opinions of healthcare providers concerning COVID-19 vaccines. By analyzing respondents’ words, qualitative research enables us to gain insights into their beliefs, experiences, attitudes, behavior, and interactions [12]. Traditional qualitative analysis involves human coders reading respondents’ words and categorizing similar responses into multiple codes or themes through an iterative consensus process [13,14].

Manually analyzing high-volume, multi-class data collected during the qualitative research process is tedious and time-consuming. Over the past decades, ML has incorporated ideas from psychology, sociology, statistics, and mathematics to enable computers to predict outcomes based on specific predictors [15]. The core functionality of machine learning is to train computers to automatically solve problems like classification using data or prior experience [16]. We used ML in two ways: 1) assist with the qualitative research process by classifying text into multiple codes; 2) use the multiple codes to predict changes in vaccine status.

1.2. Health Belief Model Constructs

The codes we used to analyze open-field comments were developed from the Health Belief Model (HBM), which can be applied to understand why healthcare providers receive or decline the COVID-19 vaccine. HBM is a psychosocial model that researchers use to understand people’s health-related behavior, including their decision to adopt or decline disease prevention measures [11]. Previous research supports the utility of HBM constructs in predicting the uptake of vaccines and measures to prevent being infected with the virus that causes COVID-19 [17].

According to HBM, whether an individual adopts a health-promoting behavior depends on five beliefs: (i) the individual’s perception of themselves as prone to disease or health risks (perceived susceptibility), (ii) their feeling of the severity of the disease (perceived severity), (iii) their challenge of taking preventative actions (perceived barriers), (iv) their view of the benefits of taking those actions (perceived benefits), and (v) their view of how well they can successfully implement the recommended health behavior (self-efficacy). Since VH individuals are hesitant about receiving a vaccine, we reasoned that the benefit and self-efficacy HBM constructs, which focus on the positives of vaccination, would not capture the health behavior of VH individuals [11]. Therefore, we focused on detecting severity, susceptibility, and barriers within the comments from nurses.

2. Materials and Methods

We developed two ML models using open-text comments provided by VH nurses in two consecutive surveys in winter-spring 2021. The first model identifies the presence of HBM constructs within the comments, and the second uses this information to predict change in VH between the two surveys.

2.1. Study Population and Data Collection

We first designed our study population to investigate the change in COVID-19 vaccine hesitancy among nurses who had participated in one of several ongoing large-scale studies. In the spring of 2020, we launched a series of surveys regarding the COVID-19 exposures and experiences of participants in the Nurses’ Health Study II and Nurses’ Health Study 3. The Nurses’ Health Study II (NHSII) was initiated in 1989 with 116,429 female registered nurses (RNs) aged 25–42 resident in 14 states. The Nurses’ Health Study 3 (NHS3) is an open cohort launched in 2010 that continues to enroll nurses and nursing students aged 18 and older, born since January 1, 1965. This cohort includes RNs, licensed practical and vocational nurses, specialized RNs, and nursing students. The NHS3 cohort was expanded to include male nurses in 2015. Altogether, the NHS studies cover more than 280,000 participants.

In April-May 2020, we invited participants who had returned the most recent primary cohort questionnaires to complete a supplementary COVID-19 survey. Exclusions to the baseline invitation, such as the lack of a valid email address, are detailed in a previous study [1]. Of 105,662 invited participants, 58,606 (55%) completed the baseline survey. Respondents were surveyed again 1, 2, 3, 6, 9, and 12 months after the initial survey. Data collection for each survey was rolled out over three weeks. We restricted the current analysis to female credentialed current, active and former nurses living in the United States who responded to the spring 2021 survey [20]. This restriction left us with 38,788 participants.

We defined vaccine hesitant (VH) as a participant who answered ‘no’ or ‘unsure’ to the winter 2021 survey question, ‘Do you plan to receive a COVID-19 vaccine?’ According to our definition, of 38,788 participants, we excluded 18,395 participants who had already been vaccinated at the time the winter survey, 337 participants who had missing data on vaccination status, and 15,814 participants who indicated they had a plan to get a COVID-19 vaccine soon, leaving a population of 4,242 VH individuals. The winter 2021 survey also provided two opportunities for respondents to express their thoughts: 1) an unprompted open-text box after the vaccine questions and 2) an open-text box at the end of the survey, prompting, ‘We are interested in learning more about your experiences during this pandemic. Please add anything else you would like to tell us here.’ Of the 4,242 VH individuals in winter 2021, we excluded 2,741 participants who didn’t write any comments in the open-text boxes, leaving a study population of 1,501 of VH participants with comments. Figure 1 depicts the flowchart of our study population.

We used this dataset of 1,501 comments as input to train an HBM ML model that predicts whether each comment belonged to any of the three HBM constructs (barriers, severity, and susceptibility) or was a non-HBM comment.

To monitor the vaccine status of survey participants three months after winter 2021, we asked the same question about vaccine status in spring 2021 survey. From the initial pool of 1,501 VH individuals surveyed in winter 2021, 96 participants who didn’t specify their vaccine status were excluded, resulting in 1,405 VH participants. Our focus for subsequent model development was narrowed down to these 1,405 individuals who participated in both our winter and spring 2021 surveys. As delineated in Figure 1, our second model is designed to predict the vaccine status of these VH individuals in spring 2021.

2.2. Machine Learning Models

In this study, we trained two ML models: 1) a model to predict an HBM construct for each comment of VH participants in winter 2021 and 2) a model to predict a change in vaccine status of VH individuals from winter 2021 to spring 2021.

To develop the HBM model, four expert annotators created a training sample set by manually annotating 300 (16% of all VH comments) comments as one of the three mentioned HBM constructs or non-HBM-related comments. Table 1 shows the guidelines for classifying the three HBM constructs, definitions, and example comments from the winter 2021 survey.

We utilized the Python programming language and its built-in scikit-learn free software machine learning library for developing and analyzing our ML models. Before training our two models, we first preprocessed the data. This involved data cleaning, segmenting each comment into words, eliminating low-variance terms (words) related to the output variables and converting words to numerical values. The details of the data preprocessing and feature selection can be found in Appendix 1.

For each training task, we trained five ML models: Random Forest (RF) [18], Multinomial Naïve Bayes (Multinomial NB), Logistic Regression, Scholastic Gradient Descent (SGD) [19], and a multi-layer perceptron Neural Network (NN) [20], and then selected the model with the best performance. The details of the data preprocessing and the features used in the ML models can be found in Appendix A.

3. Results

Our analysis of input feature reduction dimensionality by variance threshold method revealed that a subset of 705 high-variance input features is the best for training an HBM model for comments related to COVID-19 vaccination, and a subset of 430 input features is the optimal size for training a change in the vaccine hesitancy model. To provide a more comprehensive understanding of both models performance, we used weighted average recall (sensitivity), precision, F1-measure, accuracy, and Area Under the Curve for the Receiver Operating Characteristic curve (AUC-ROC).

Recall, also known as sensitivity and true positive rate, is a metric that measures the proportion of accurate positive predictions among all possible positive predictions. Recall ranges from 0 to 1 and high recall indicates that the model can identify most of the positive instances and is calculated using the equation below.

$$recall=\frac{True Positive}{True Positive+False Negative}$$

We explain the details of each model in the following sections. A precision measure calculates how many correctly positive predictions were made and is calculated using the equation below:

$$precision=\frac{True Positive}{True Positive+False Positive}$$

F1-measure is a harmonic mean of precision and recall and ranges from 0 (prediction failure) to 1 (perfect prediction). A high F-measure indicates both good precision and recall, meaning the model has a good balance between minimizing false positives and false negatives.

$$F1-measure=2\ast \frac{Precision\ast Recall}{Precision+Recall}$$

Accuracy is a common metric used to evaluate the performance of a classification model. It measures the proportion of correctly classified instances out of the total instances in the dataset.

$$accuracy=\frac{True Positive+True Negative}{True Positive+False Positive+False Negative+True Negative}$$

The Area Under the Curve (AUC) of the ROC curve is a scalar value that quantifies the overall performance of a classification model. A model with no predictive power would have an AUC of 0.5. This is equivalent to a random guess. A model with perfect predictive power would have an AUC of 1. A steeper ROC curve generally indicates better model performance.

3.1. HBM Prediction

Table 2 depicts the performance of the models we trained to predict the label for the unlabeled dataset (the rest of data that we didn’t use for HBM annotation). The label was any of the three constructs of HBM (barrier, susceptibility, and severity) or a non-HBM-related construct. The NN model outperformed other models with 82% accuracy. As Figure 7 (b) shows the NN model demonstrated robust performance with an area under the ROC curve (AUC-ROC) of 91%, indicates its high discriminatory ability. Figure 2, Figure 3, Figure 4 and Figure 5 show the most influential meaningful words in detecting HBM constructs in all comments from VH individuals.

The results of our subsequent data analysis of the words in VH comments for each HBM construct are shown in Figure 6. We found that 35% of the comments from the VH nurses indicated perceived barriers to vaccination and that this was the most frequently expressed belief type. Our ML analysis on HBM constructs also suggested more specific details about these barriers; the most significant barriers to receiving the vaccine for the VH participants appear to be side effects and allergic reactions, with other concerns regarding pregnancy and fertility that were potentially related to the lack of research on the vaccine side effects within these domains.

The HBM classification also revealed that 17% of VH comments mentioned a perceived lack of severity of COVID-19. The most related topics under this category tended to mention high recovery rates from contracting COVID-19. An additional 16% of VH comments mentioned low susceptibility to contracting COVID-19. This category mentioned words suggesting respondents felt they already had a robust immune system, such as ‘supplements,’ ‘boost,’ and ‘diet.’

3.2. Results for Vaccine Status Prediction

The model used the comments from VH individuals in winter 2021 to predict vaccine status of the VH in spring 2021. After expressing vaccine hesitancy in the winter 2021 survey, 40% of nurses received a COVID-19 vaccine by spring 2021, while 60% remained unvaccinated. Table 3 shows the performance of the trained models for predicting the vaccine status from VH comments. The best accuracy achieved was 89% using a NN. Figure 7 (a) shows the AUC-ROC curve of the model performance.

4. Discussion

This study used open-text comments collected in winter 2021 to predict vaccine status and hesitancy three months later. Throughout this time, the vaccines were accessible only under an Emergency Use Authorization and not yet mandated for nurses to be vaccinated. The winter 2021 survey results showed that 11% of the participating nurses expressed vaccine hesitancy and 4% wrote at least one comment. In spring 2021, 60% of the VH in winter 2021 remained hesitant three months later. Our results suggest that while most early VH may be due to perceived barriers, there is a small cohort for whom a belief in low COVID-19 severity or low personal susceptibility will linger. Our findings extend previous studies using qualitative coding to analyze COVID-19-related surveys [24–26].

We found the most commonly occurring HBM construct within the comments was perceived barriers, suggesting concerns over the tangible and intangible costs associated with getting the vaccine were prevalent among VH nurses. Specific costs mentioned included side effects and allergic reactions, indicating opportunities to develop messaging to allay these concerns. Pregnancy and fertility concerns were also frequently mentioned among comments assigned to this construct, likely owing to the lack of evidence regarding the vaccine’s safety among pregnant women and the messaging uncertainty among obstetricians [27].

This study takes the unique approach of using ML to categorize open-text comments into HBM constructs. Both HBM classifier’s 82% accuracy rate to predict HBM constructs and 89% accuracy rate for predicting vaccine status from text reveal that implementing ML has the potential to automate the qualitative research project. In addition, the area under ROC curve for both models exceeding 80% indicates robust model performance, signifying models’ effectiveness in predicting the targets compared to random guessing.

Our methods would also be helpful for future research augmenting ML with health behavior models. Such theoretical models of human behavior boast clear demarcations between constructs, necessitating corresponding ML methods to identify distinguishing features (e.g., words). Our analysis of feature selection revealed that a subset of high-variance words from each comment, rather than the entire comment, leads to improved performance in machine learning models. Additionally, we observed that excluding words representing two classes equally enhanced model performance. Similar approaches will likely yield better performance of ML methods that adopt a theoretical framework.

In our study, among several classical ML models we tested, the neural network model surpassed all other models, achieving the highest performance in both the HBM model and vaccine status prediction models. We also discovered that applying variance threshold for input feature dimensionality reduction helped to optimize the ML models training.

Readers should bear in mind the study’s limitations when drawing conclusions. First, our findings only apply to English-speaking nurses in the US, as only English text was used to develop the model. Additionally, the cohort primarily consisted of individuals of white ethnicity hence our ML model has not been trained to recognize how other racial groups feel about the COVID-19 vaccines. Moreover, not every VH nurse responded to the open-text questions used in the analysis. Lastly, in considering susceptibility, this category could potentially carry a valence. For instance, a comment could mention that they are choosing to not receive a vaccine because they are not susceptible to COVID. Opposingly, a comment could mention that they are choosing to receive a vaccine because they are susceptible to COVID. Since our study population is VH individuals, we chose to focus our analysis on the former valency of the susceptibility category. By expanding our study population to all individuals, future studies can consider both valences. Nonetheless, we were able to obtain a high accuracy rate. Despite these limitations, we believe that ML has the ability to streamline qualitative research. As the field of ML advances with newer models, researchers in other fields have an opportunity to automate tedious tasks and extract useful information from their data.