Comparative Statistical Analysis between Neutralization Assays in Vaccine Immunogenicity Assessments

1. Introduction

The emergence of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) necessitated the rapid development and deployment of vaccines and therapeutic agents. A critical component in evaluating the effectiveness of these interventions is the use of neutralization assays, which measure the ability of antibodies to inhibit virus infection. By measuring the neutralizing activity, researchers can determine the potency and duration of immune responses elicited by the vaccine candidates, guiding modifications and improvements in their formulation[1,2]. Additionally, understanding variations in neutralization efficacy against emerging variants of the virus is vital for ensuring long-term protection and effectiveness of the developed vaccines and therapeutics[3,4,5,6].

Traditional neutralization assays with live viruses directly measure antibody effectiveness against actual pathogens and are considered the gold standard for evaluating neutralizing antibodies[7,8]. However, live virus neutralization assays (LVNAs) pose several challenges, such as biosafety issues and the need for high-level containment. To overcome these limitations, pseudotyped virus neutralization assays (PVNAs) use non-pathogenic viruses that express the SARS-CoV-2 spike protein, allowing for a safe simulation of viral entry. Core viruses like vesicular stomatitis virus (VSV) and lentiviruses (e.g., HIV-1) are commonly used, making PVNAs suitable for biosafety level-2 (BSL-2) laboratories, unlike live virus assays that require BSL-3[9,10]. PVNAs enable high-throughput screening and adaptability for various strains, which is important for rapidly evolving pathogens.

It is essential to recognize that discrepancies may exist between PVNAs and LVNAs. These differences can stem from variations in the viral life cycle, entry efficiency, and cellular tropism. While PVNAs are useful for initial screening and mechanistic studies, they should be validated with LVNAs to ensure relevance to natural infections. Understanding the correlation between the two is crucial for validating PVNAs as a reliable proxy for LVNAs in vaccine and therapeutic development. Previous studies have shown varying degrees of concordance[8,11,12,13,14,15,16]. Most of these studies analyzed the correlation based on small sample sizes, and used serum samples from COVID-19 convalescents instead of vaccinated individuals. Additionally, only several of them evaluated the correlation across different variants[8,15,16].

This post-hoc analysis aimed to evaluate the correlation between a PVNA and an LVNA in a phase I/II clinical trial that evaluated a bivalent protein-based COVID-19 vaccine, providing insights into their relative performance and reliability.

2. Materials and Methods

2.1. Study Design and Participants

This comprehensive post-hoc analysis examined data from a Phase I/II clinical trial of vaccine interventions. Phase I assessed the safety of different dose levels of a vaccine in healthy volunteers, while Phase II evaluated its effectiveness in a larger, diverse population, monitoring safety. Participants who had neither been infected with SARS-CoV-2 nor received any SARS-CoV-2 vaccines were included. They all received two doses of SCTV01C as part of their primary series vaccination. SCTV01C is a recombinant bivalent vaccine comprised of the trimeric spike extracellular domain (S-ECD) of SARS-CoV-2 variants Alpha and Beta, and adjuvanted with SCT-VA02B, a squalene-based oil-in-water emulsion. Immunogenicity was evaluated pre-and post-vaccination by detecting the titers of neutralizing antibodies. The primary results of this clinical trial have been published[17,18]. This analysis aimed to uncover correlations between neutralizing antibody results from LVNA and PVNA across different SARS-CoV-2 variants.

2.2. Neutralization Assays

Neutralizing antibodies against SARS-CoV-2 Alpha, Beta, and Delta variants were measured using LVNA and PVNA.

SARS-CoV-2 Microneutralization Assay

The live virus neutralizing antibodies were measured using a microneutralization assay (MNA) and have been described previously[17]. Sera were diluted serially in duplicate, starting at 1:8, and then mixed with live virus for incubation. This mixture was then tested on susceptible cell cultures for signs of viral infection. The neutralization titers were defined as the reciprocal of the highest dilution that protected 50% of wells (MN50) and were calculated using the Reed-Muench equation.

SARS-CoV-2 Pseudotyped Virus Neutralization Assay

The establishment and validation of the SARS-CoV-2 pseudotyped virus neutralization assay have been reported[9,19]. A pseudovirus was produced using a VSV pseudovirus production system. Sera were diluted serially, starting at 1:30, and then mixed with pseudotyped virus, followed by incubation. The mixture was then incubated with target cells, and the virus infecting the target cells was measured by detecting the expression of luciferase. The half maximal effective concentration (EC50) titers were calculated based on uninfected cells as 100% neutralization. Reciprocal EC50 geometric mean titers (GMT) were also determined.

2.3. Statistical Analysis

Among all the serum samples (before or after baseline) from the analysis population, those with both LVNA and PVNA test results under the same SARS-CoV-2 variant (Alpha, Beta, or Delta) were selected for analysis. Moreover, the results of the three SARS-CoV-2 variants were analyzed separately. The statistical analyses were conducted using SAS software (version 9.4). In calculating specificity, sensitivity, and accuracy, antibody titers against SARS-CoV-2 equal or greater than the lower limit of quantitation were labeled as “Positive,” while those below were labeled as “Negative.” The antibody titers of LVNA were taken as the reference (“true”) results, and the misclassifications were counted and displayed in an error matrix table. The Pearson correlation coefficient and a linear regression model were used to measure the strength of the relationship between PVNA and LVNA. All antibody titers were log-transformed to the base 10 before calculation.

To evaluate the agreement between PVNA and LVNA, a Bland–Altman plot[20] and a Kernel density plot were plotted. The Bland–Altman plot is a scatter plot of the mean PVNA and LVNA at each measurement point, along with their differences. The PVNA and LVNA were log-transformed to the base 10 before calculating the mean and difference. The 95% limits of agreement (LOAs) and the maximum acceptable difference (MAD) were calculated. The LOAs were constructed as a V-shaped limit[21], and the MAD was set to 0.5 times the LVNT. If the observed PVNA-LVNA difference is below the MAD value, it is considered that the difference has no significant biological effect. The Gaussian kernel is chosen to plot the kernel density plot, which describes the probability distribution of the fold increase relative to the baseline of antibody titers after baseline. The fold increase was log-transformed to base 10 when plotted.

3. Results

3.1. The Neutralizing Antibody Titers

Descriptive analysis was performed on all the serum samples, categorized by the sampling visit times, assay methods, and corresponding variants. The GMT and its 95% confidence intervals were calculated for each category, and the results are listed in Table 1. It can be seen that at baseline, almost all the antibody titers were below the lower limit of quantitation. At the first sampling visit after the second vaccination, the antibody titer levels showed a significant increase compared to baseline, followed by a gradual decrease over time. However, the neutralizing antibodies increased significantly 365 days after vaccination. The possible reasons might include known or unknown SARS-CoV-2 infection and close contact with COVID-19 individuals during the SARS-CoV-2 epidemic due to the decline of neutralizing antibodies and the emergence of new variants.

3.2. The Error Matrix

The number of serum samples tested for the Alpha, Beta, and Delta variants was 324, 324, and 505, respectively, with the results shown in Table 2. It can be seen that the PVNA has a very high sensitivity for LVNA. In detecting all three variants, only one LVNA-positive sample was negative in the PVNA test. The specificity of PVNA for LVNA was also good, with a specificity greater than 90% for all three variants. The accuracy of PVNA for LVNA was 98.8%, 99.1%, and 94.3% for the Alpha, Beta, and Delta variants, respectively, reflecting the high consistency in the test results.

3.3. The Correlation between PVNA and LVNA

The Pearson correlation coefficients between PVNA and LVNA were 0.9614, 0.9517, and 0.9072 for the Alpha, Beta, and Delta variants, respectively, indicating that PVNA and LVNA have a strong positive correlation. Figure 1(A) and Figure 1(B) show that almost all the points are distributed near the regression line, indicating a strong linear relationship between PVNA and LVNA after log transformation. As shown in Figure 1(C), although most points were distributed near the regression line, there were some points where the titers of LVNA were below the lower limit of quantitation, while those of PVNA remained high. This resulted in a weaker correlation between LVNA and PVNA in the Delta variant compared to the Alpha and Beta variants.

3.4. Bland-Altman Analysis

The results of the Bland–Altman analysis are shown in Figure 2. Almost all the points in the figures were distributed close to the regression line, and the difference did not increase as the mean increased. Additionally, the angle between the two LOA lines was very small, indicating good agreement between PVNT and LVNT. In Figure 2(A) and Figure 2(B), only a few points lay outside the range of the MAD, with very few outliers. However, in Figure 2(C), several points appeared above the MAD. Compared to Figure 1, it can be seen that these points correspond to the serum samples where titers of LVNA were below the lower limit of quantitation, but those of PVNA still had a detectable value. Aside from these points, the rest showed high agreement between PVNA and LVNA.

3.5. The Probability Distribution of the Fold Increase Relative to the Baseline of PVNA and LVNA

As shown in Table 1, the GMT of PVNA was higher than that of LVNA. Considering the differences in assay methods, the fold increase relative to baseline was used to analyze the agreement between PVNA and LVNA. Figure 3(A) and 3(B) showed that the distribution curves of PVNA results overlap almost entirely with those of LVNA, indicating good agreement between PVNA and LVNA for Alpha and Beta variants. In Figure 3C, the curves of PVNA and LVNA were similar in shape but do not overlap, with the relative baseline fold increase of PVNA being slightly lower than that of LVNA. The agreement of fold increase relative to baseline between PVNA and LVNA for the Delta variant was slightly lower than that for the Alpha and Beta variants.

4. Discussion

Antibody detection and quantification methods are crucial for assessing immune response post-infection or vaccination. While LVNA directly evaluates neutralizing capability in an infectious context, its application faces several challenges, such as stringent safety protocols, regulatory constraints, and variability in viral strain availability[22]. These limitations hinder the scope and frequency of experimental studies, ultimately affecting the efficiency of vaccine and therapeutic development efforts[23]. In contrast, PVNA demonstrates notable advantages, including increased safety, accessibility, and versatility in testing for various viral threats, such as SARS-CoV-2[24], HIV[25], HPV[26], Influenza[27], and others. These characteristics illustrate the growing preference for PVNA in virology, providing a vital tool for advancing our understanding of viral infections and immune responses[28,29]. This has been accepted as an assay for the assessment of immunogenicity endpoints in the FDA guidance for vaccines to prevent COVID-19[30]. Understanding the correlation between LVNA and PVNA is important for evaluating whether pseudotyped assays can effectively replace live virus assays in viral testing. This knowledge ensures both the practicality of testing methods and the safety involved, as working with live viruses carries significant risks. Our findings indicate a significant alignment between these methods, while also highlighting essential nuances for researchers in vaccine development and therapeutic evaluations.

We employed various statistical methods to analyze the correlation between LVNA and PVNA. The consistency observed across these different approaches provides a robust foundation for understanding the relationship between the two methodologies. As shown in Table 2, the analysis of PVNA versus LVNA for SARS-CoV-2 variants, Alpha, Beta, and Delta, demonstrated that the sensitivity and specificity of PVNA exceeded 90% for all variants, indicating its potential as a robust and reliable tool for assessing neutralizing antibody responses in populations exposed to SARS-CoV-2. Furthermore, the accuracy rates recorded were 98.8% for Alpha, 99.1% for Beta, and 94.3% for Delta. These results underscored the effectiveness of PVNA in distinguishing neutralization capabilities against these variants. The Pearson correlation coefficients of 0.9614 for Alpha and 0.9517 for Beta demonstrated that PVNA could effectively replicate the neutralization dynamics of live viruses in controlled settings. Conversely, the correlation for Delta (0.9072) was a little weaker, suggesting potential variability in neutralization capabilities, which warrants further investigation into the impact of specific mutations in this variant. Bland-Altman analysis further supports the reliability of PVNA, showing good agreement with LVNA results. We visualized any systematic bias and identified outliers by plotting the difference between LVNA and PVNA results against their average. Our analysis revealed minimal bias and narrow limits of agreement, indicating that the discrepancies between the two methods are generally small and random. Notably, while the geometric mean titer (GMT) of PVNA was higher than that of LVNA for the Delta variant, the slightly lower fold increase in titers of PVNA may indicate that certain mutations in the spike protein of Delta could reduce the efficacy of neutralization. Future research should continue to refine these methodologies to enhance their predictive accuracy and broaden their applicability in vaccine development and evaluation.

It is important to interpret the correlation between PVNA and LVNA carefully. While our findings support PVNA's reliability, differences in methodology - such as the use of engineered particles versus live viruses - can affect neutralization effectiveness. These variations may be due to differing biological testing contexts and immune responses. PVNA utilizes recombinant, replication-deficient viruses engineered to express SARS-CoV-2 spike proteins on their surface. The ability of antibodies to neutralize viruses may not fully translate from pseudotyped to live viruses due to the complex interactions present in live viral infections[8,14,31].

There are some limitations in our study. Firstly, neutralizing antibodies against Omicron were not evaluated, and the impact of new mutations on the correlation between these methods remains to be investigated. Secondly, the LVNA and PVNA were not performed by the same experimenter, which may introduce additional errors due to the differences in procedure and data reading.

5. Conclusion

In conclusion, while LVNA provides a precise measure of vaccine-induced immunity, PVNA offers benefits in safety, cost, and scalability. It is important to understand the strengths and limitations of both methods for effective vaccine evaluation, especially with new SARS-CoV-2 variants. Future research should aim to enhance PVNA techniques to better represent natural infections and improve accuracy for various viral strains. By combining efforts, both approaches can accelerate the development of effective vaccines against evolving pathogens.