Measuring the Vaccine Success Index: A Framework for Long-Term Economic Evaluation and Monitoring about the Case of Rotavirus Vaccination

1. Introduction

The success of a vaccine is expressed as its vaccine effect (VE). This is referred to as ‘vaccine efficacy’ when assessed under the strict conditions of a double-blinded randomized controlled trial (RCT) evaluating vaccinated and unvaccinated subjects over a fixed period [1], and as ‘vaccine effectiveness’ when assessed in uncontrolled conditions comparing vaccinated and unvaccinated groups in real-world practice [2]. VE may change over time and may decrease, described as vaccine waning. Different types of waning can be identified [3]. In classical waning, the vaccine inoculum does not activate a sustainable immune response in the host, and this can be remedied by a vaccine booster dose. Another waning process is related to the vaccine’s biological activity, producing partial antigen activation, and resulting in lower VE over time after multiple exposures to the pathogen. Such leaky or partially responding effects have been described in studies analyzing influenza vaccines introduced in recent years [4,5,6]. A third type of waning, which differs from the previous two in that it is not caused by the vaccine, may appear with suboptimal introduction of a new vaccine. This type of waning, referred to in this paper as ‘strategic waning’, can only be detected by long-term measurement of VE [7], and is the focus of this paper. This analysis proposes an approach to measuring the success of a vaccination program from an economic perspective across the whole population, including any strategic waning effect. Epidemiological assessments of infectious diseases and vaccination are made at the level of comparative groups and use the term ‘effectiveness’, whereas evaluations at the level of the whole population use the term ‘impact’ assessment [8]. Health economic evaluations have not conventionally made this distinction; however, it could help to understand that vaccination may have a broader and longer economic impact on the whole population extending beyond specific comparable at-risk target groups [9,10]. Furthermore, a sub-optimal start to a vaccination program may affect the long-term impact of the vaccination program on the whole population. This paper illustrates this potential effect and highlights the importance of evaluating a vaccination program in a broader context than conventional. Most of the analyses and results presented here are based on rotavirus vaccine introduction in Belgium using data from the RotaBIS study. Many of the results have already been published elsewhere [7,9,11,12]. This paper builds on previous publications by making the link between effectiveness and impact and showing how the success or failure of a new vaccination program can be measured using a new ratio calculation, called the success index. This helps to assess the economic long-term value achieved with the vaccination in complex environments. The paper uses cost-effectiveness and cost-impact analysis to obtain results that should help healthcare payers assess whether the investment in a vaccination program was successful [13].

2. Materials and Methods

Background

This analysis concerns rotavirus infection and its vaccination in a high-income country, Belgium. Rotavirus infection is distinctive in that it is seasonal in the northern hemisphere, spreading during each winter period from around January to March/April, and affects children up to 5 years old, causing severe diarrhea with a risk of dehydration [14]. Different sources of this infection exist, but the primary source is infants aged 3 to 14 months who spread the infection across the whole at-risk grouMore exposure to infection increases the immunity level against the disease [15]. The infection is contagious, with a primary reproductive number of R₀ estimated at around 9 in normal conditions [16,17]. The disease spread as a function of age has a clear Weibull distribution pattern with a mean around 15 months of age and a long decreasing tail to the right up to 60 months of age [14]. Rotavirus infection has mainly been studied by the rate of cases leading to hospitalization, with less measurement of infection rates at the primary healthcare or family care level as laboratory tests are not systematically conducted. Two vaccines are available in high-income countries, a two-dose vaccine (Rotarix, GSK Biologicals) and a three-dose vaccine (Rotateq, Merck), and should be given at the age of 6 weeks for the first dose with a 4-week interval for subsequent doses [18,19,20]. As reported in clinical trials, vaccination provides a very high response rate in reducing severe cases [21,22]. However, the vaccination should be given prior to 32 weeks of age as it may have a very low risk of the severe adverse effect of intussusception [23,24]. There is, therefore, no option to introduce a catch-up vaccine strategy to vaccinate everyone at risk at once. Consequently, every newborn needs to be vaccinated, and the time required to cover the whole at-risk group (children up to 5 years old) with vaccination may be at least 5 years. The possibility of vaccine waning over time has been suggested, based on comparing the first- and second-year efficacy data [25]. When introduced in a child population in the defined age group (aged 6 weeks to 8 months) the vaccine diminishes infection in the primary source (infants aged 3–14 months) and consequently causes an indirect protective effect amongst the unvaccinated individuals in the population, which also reduces the potential for creating secondary sources of infection in the same target group, being infection sources developed as a consequence of the primary source. So the primary source is not only the cause of a direct infection but also the cause of secondary infections [11]. If the vaccine substantially reduces the primary source of infection by achieving a high coverage rate at start of the vaccination program, it can also reduce the secondary sources of infection, thus causing an important additional indirect effect. If the vaccine coverage is lower, and/or if the timing of the program introduction is sub-optimal in relation to the disease seasonality, it allows a greater manifestation of secondary sources of infection that are not influenced by the vaccine's direct effect. The level of secondary infection sources affects the vaccine's effect over time and the endemicity level of rotavirus in the whole population. Increasing vaccine coverage has an immediate impact only on the initial primary source, with a lesser effect on secondary sources of infection that develop into the new primary source of infection in the population over time.

Economic Assessment

The economic value assessment of rotavirus vaccination in the short to long term rests on two pillars, data and modeling, which help to explain the observations and allow credible simulations.

Data

The data were collected from the RotaBIS study, which was initiated in 2007, a year after the vaccine was introduced and partially reimbursed by the Belgian authorities in November 2006 [26,27,28]. Data on disease-specific hospitalizations were retrospectively collected for 2005 and 2006, before the vaccine's introduction. The same information was subsequently gathered annually for 13 years (2007 to 2019) from 11 hospitals that were representative of the different parts of the country. The following data were assembled for each event, in addition to the test result and date for rotavirus detection: the date of hospitalization; the specific age when the disease occurred; sex; duration of hospitalization; and nosocomial acquisition. The entire protocol of the study has been reported elsewhere [29]. The information relevant to the present analysis is summarized in Table 1, showing the numbers of disease-specific hospitalizations by age and year reported over a total period of 15 years (the pre-vaccination years of 2005 and 2006 are reported as average values for the two years combined).

Modelling

The model must replicate the observed data (blue line in Figure 1) and include those variables that affect the shape of the observed curve using direct and indirect vaccine effects [7]. To facilitate the analysis, the model splits the observation period into two consecutive linked periods, using a different model type for each period (Figure 1). Full details of the models, including sensitivity analyses, have been published elsewhere [7].

The first vaccination period is the vaccine uptake period, which can last 5 to 7 years until a new infection equilibrium has been reached in the target group of children aged ≤5 years. For this period, the model uses a time-dependent regression equation to characterize the shape of the curve, in which different forces influence the regression line, simulating the number of disease-specific hospitalizations observed per year. Two main forces were identified, each of which combines several components. The first force defines the direct vaccine effect with components for effectiveness, coverage, and waning. The second force represents the indirect effect of the vaccine, with components affecting the herd effect and secondary sources of infection. This uptake period is here defined as the one that measures vaccine ‘effectiveness,’ with a supposedly stable new infection equilibrium reached in the target group at the end of the period.

The vaccine uptake period is followed by a post-uptake period, in which the dynamic spread of the infection is simulated using a time differential compartmental equation with susceptible, infectious, and recovered (SIR) groups linked by transition rates, and simulating the observed biennial disease peaks over time using a Hamer model design [30]. The frequency and height of the peaks depend on the entry conditions at the end of the vaccine uptake period. These entry conditions include the remaining infection rate in the population, the maintained vaccine coverage rate with its net effect, the susceptible group (newborns) entering at any given time point, and the contact matrix of the at-risk population. It is important to note that the initial primary source of infection pre-vaccination will have shifted in the post-uptake period to an older age group. This shift develops during the vaccine uptake period when the vaccine coverage and the timing of initiating the vaccination program are not optimal. The whole period of vaccine uptake plus post-uptake is defined as the period over which the vaccine ‘impact’ assessment is calculated.

Cost-Effectiveness and Cost-Impact Analysis

The economic evaluations have an identical basic formula to assess the two measures of interest: the cost-effectiveness analysis (CEA) and the cost-impact analysis (CIA). CEA references the vaccine uptake period using the modeled or observed data as input until the post-uptake period is reached. CEA evaluates the initial target vaccination population, considering the effectiveness of direct and indirect effects of the vaccine, such as the positive herd effect and the opposing effect of secondary sources of infection in unvaccinated individuals, and compares the results with a historical group that is unvaccinated [31]. The CIA covers a more extended period than the CEA, including the uptake and post-uptake periods and the whole vaccinated and unvaccinated population in which the vaccine has direct and indirect effects (the impact of the beneficial herd effect will be less than in CEA and impact of the detrimental secondary infection source effect will be greater) again over time. The accumulated results are compared with the situation before initiating the vaccination program (pv) [10]. The formula for CIA is as follows:

$$\begin{gathered} CIA={\displaystyle \frac{∆C}{∆E}} \\ ∆C=\mathrm{}{(C}_{pv}-(C_{uv}+C_v)) \\ ∆E={(E}_{pv}-\left(E_{uv}+E_v\right)) \end{gathered}$$

∆ = difference; C = cost; E = health effect often expressed in quality-adjusted life-years (QALYs); pv = pre-vaccination; uv = unvaccinated; v = vaccinated

The negative indirect effect of the vaccination on the whole population is added to the evaluation as secondary sources of infection in older age groups that develop into new primary sources of infection if the vaccination program initiation is not optimal. This evaluation method for CIA refers to the impact assessment in epidemiology, as presented by Hanquet et al. [8], applied here as an economic assessment.

Input Data

The input data used to estimate the cost and QALY loss due to rotavirus hospitalizations are presented in Table 2, based on data from Belgium. The cost data are those used when the vaccine was launched in 2006 and received its reimbursement price in Belgium, which has not changed.

Input values for the key variables that define the shape of the curve during the vaccine uptake period are presented in Table 3.

Output Data

The output obtained is the incremental cost-effect ratio (ICER) achieved, with ‘effect’ defined as ‘effectiveness’ or ‘impact’ measurement for CEA and CIA respectively. Scenario evaluations are performed comparing the ICER of the uptake period (7 years) with the ICIR obtained of the uptake + post-uptake period (15 years), observed data. The ICIR obtained of the observed data is then compared with the ICIR of the optimal launch and the ICIR of the worst-case launch. Discounting cost or health gains was not applied, because the present analysis is concerned with assessing the optimal strategy for vaccine launch and not with assessing the vaccine's value.

The output for the optimal launch and the worst-case vaccine launch scenarios used the previous two models, where the vaccine coverage rates were adjusted in each scenario (86% from the start in the optimal launch scenario and 40% in the worst-case scenario) with a slightly lower VE (0.86 instead of 0.95). The lower VE is justified by the potential presence of the two vaccine waning processes over time (reduced immune response, leaky vaccine). Figure 2 illustrates the decrease in hospitalizations simulated with the model programs for an optimal vaccine launch and a worst-case scenario of a poor vaccination launch, together with the observed data.

The Success Index

The results of CEA and CIA do not reveal whether a vaccination program has been successful over time, with ‘success’ meaning that the vaccination is maintaining control of the infection spread, indicated by very low rates of disease-specific hospitalizations in the long term. Lower CEA results are often considered to have reached better outcomes. However, CEA is mainly performed to obtain a value assessment related to the acceptable vaccine price; therefore, a low CEA could also result from a low vaccine price and not from reasonable long-term control of infection spread achieved by the vaccination program. Reasonable control of infection spread would have the effect of reducing the frequency and height of the biennial peaks in the post-uptake period. There are, however, two situations in which those post-uptake peaks could be reduced. One results from optimal vaccine introduction with a high vaccine coverage at the start, inducing a high indirect effect with control of primary and secondary sources of infection resulting in peaks of limited size and frequency in the long term. The other could result from deficient vaccine uptake, resulting in the continued dominance of the initial primary source of infection and consequently no clear manifestation of secondary sources of infection in the long term. The two situations can be distinguished by considering the ratio of the results of CIA over CEA, here called the success index. A successful vaccination launch will have a ratio close to 1 with a low CEA result. A poor vaccination launch will also have a ratio close to 1, but with a high CEA result. Intermediate results indicate that the vaccination program was not a failure but could have been more successful with an optimal vaccine launch. The ratio will be close to 1.5, combined with a higher CEA result than projected for the optimal launch. There is, therefore, a maximum ratio for CIA over CEA, dependent on the level of attenuation of the primary source of infection by the vaccination program.

3. Results

ICER Results

The cost-effectiveness results for the uptake period compared with no vaccination are shown in Table 4.

The results shown in Table 5 compare hospitalization days for the whole uptake period including post-uptake data with results for no vaccination. The ICER result from this CIA differs from that calculated in the previous CEA (Table 4) due to the appearance of the secondary hospitalization peaks at 9 and 11 years post-vaccine introduction, which negatively impacts VE over time.

The Success Index and the Scenario Analyses

Table 6 shows the success index ratio calculated from the CEA results (Table 4) and the CIA results (Table 5). Table 6 also presents the simulation results for the two scenarios of optimal vaccine launch and worst-case vaccine launch. Figure 3 shows the simulation results for the success index ratio expressed as a function of the CEA value calculated over a pre-specified evaluation period (defined in this analysis as 15 years). The results follow a lognormal outline distribution, illustrating the success and failure areas for this setting when the threshold for the CIA/CEA ratio has been set at 1 (CEA_max=15,091€ for S; CEA_min=60,296€ for F). The results for the optimal launch scenario, the worst-case launch scenario and the observed Belgian data are plotted as point values; the optimal launch scenario falls into the success ‘S’ area of the outline, the worst-case launch scenario falls into the failure ‘F’ area, and the observed data from Belgium fall into an intermediate area (Figure 3).

4. Discussion

The analysis presented here illustrates a potential approach for the assessment of preventive health programs, emphasizing the importance of establishing clear objectives regarding cost and health outcomes over defined timeframes at the start of a vaccination program. The proposed success index offers a way to evaluate the performance of the program in real world over time. Initially, the rotavirus vaccination campaign targeted of what was perceived as a minor health concern in high-income countries. Disease elimination was anticipated through progressively expanding vaccine coverage, and some models predicted that success when the vaccine was launched in 2006 [35,36]. Early vaccination outcomes exhibited notable reductions in rotavirus hospitalizations within the first two years, yet subsequent declines plateaued [26]. The absence of systematic, ongoing health gain monitoring meant that new, biennial disease peaks appearing some years after vaccine introduction, reflecting strategic waning of the vaccination, may not always be reported. The RotaBIS study in Belgium addressed this lack of monitoring by initiating a routine annual data collection system once the vaccine was approved and reimbursed [29]. The study was able to maintain the data processing for several years into the post-uptake period because it was collecting data recorded in routine practice and thus required a limited budget assignment. Finally, rigorous data analysis was crucial for identifying anomalous trends and understanding underlying phenomena. The use of comparative analyses with subtly different scenarios provided valuable insights, informing interpretations of observed data anomalies. This required the creation of two evaluation periods in the analysis, the use of two models, outcome measures assessed over two periods, and the development of a new impact calculation, familiar in the epidemiology world but relatively new in the health economics arena. The approach illustrated the necessity of understanding the data assembled, the effect of different infection sources on disease spread, and the effect on the vaccine impact of a sub-optimal introduction. The sensitivity analyses of worst-case and optimal vaccination launch scenarios were useful because different countries were found to be close to the results projected in both scenarios. The United Kingdom (UK) and Finland indicate what might be expected with a near-optimal rotavirus vaccine launch, compared with the sub-optimal launches in Spain and Ireland [37,38,39,40]. The present analysis proposes an approach to investigating whether a vaccination program was successful using the success index ratio.

This comprehensive analysis illustrated the importance of considering impact evaluations in economic assessment, as epidemiology did a few decades ago [8,41]. It could be argued that CEA will capture the impact from short to long duration in any case. However, it is important to make the distinction between effects in populations at risk compared with the whole population. The effects in the wider population, not the population targeted by vaccination, is cause of the new primary source of rotavirus infection (older children who in the pre-vaccination period had been a secondary source of infection) that appeared in the post-uptake period when the vaccination was not optimally introduced at start. This new primary source will not be easy to target because the rotavirus vaccine does not directly reach the new source, as these children are too old to receive the vaccine. Any vaccination campaign introduced across a population may affect others besides those at-risk groups for whom vaccination is intended. Broader consequences may appear, which need to be captured in economic evaluations. It was an option to stop the RotaBIS monitoring study after eight years when the new infection equilibrium in the target population appeared to have been reached. Fortunately, the study continued to collect data over a longer period, which helped to construct a better explanation for the plateau reached during the uptake period, approximately three years after the vaccine introduction. The study's emphasis on impact evaluations within the economic assessments of vaccines reflects a paradigm shift in health economics, akin to methodologies long employed in epidemiological research. This approach, exemplified by the rotavirus vaccination case study, acknowledges the broader societal impacts of vaccination beyond individual health gains. Sensitivity analyses exploring various vaccination scenarios further contextualized the findings and provided insights into optimal vaccination strategies.

The present analysis developed the success index because it may be useful for healthcare decision-makers to have an unequivocal measure of the success of a vaccination program. The results observed in Belgium during the first two years after rotavirus vaccine introduction were considered a great success, as the country was the first high-income country to introduce the new vaccine systematically. However, our modelling analyses, supported by experience with rotavirus vaccine introduction in other countries, indicate that better outcomes could have been obtained if more detail about the infection spread and disease burden had been available and applied to inform the implementation and timing of the vaccine introduction in Belgium. Our results may be useful for any new vaccine in current or future development [42].

Vaccines and infections need to be assessed individually to take account of their unique characteristics. Nevertheless, the assessment framework presented here has broad applicability. The evaluation approach, focusing on community infection control processes, transcends traditional cost-effectiveness analyses, offering a more comprehensive understanding of the value of a vaccination program expressed through its success index score. Although developed with specific reference to the case of rotavirus vaccination, the principles and methodologies outlined in the present study are relevant for assessing the clinical and economic implications of future vaccine introductions using such a score.