Lessons Learned from Modeling COVID-19: Steps to Take at the Start of the Next Pandemic

1. Record Infections, Hospitalizations, and Deaths per Region

Accurate models are essential for formulating policies that can safeguard both lives and livelihoods. However, developing precise forecasting models is an arduous task without reliable data. Collected data should be made public in a trusted location and multiple standardized formats such as CSV files, spreadsheets, and PDFs. During the COVID-19 pandemic, numerous organizations including the Covid Tracking Project [2], Johns Hopkins University, etc., released U.S. data in a systematic manner that was both machine and human-readable. However, how those numbers are recorded should be standardized across regions and countries for uniformity and handled with a professional level of quality control. A lack of standardization can result in data misinterpretation and incorrect conclusions. For example, the number of positive tests is often confused with the number of infections, which are not the same. Additionally, modern methods such as wastewater metric data can help in more accurately determining the proportion [3] of viruses in a population. Moreover, it is important to record numbers “consistently”, especially numbers for hospitalization and deaths that may be more accurate. During this last pandemic, there have been many inaccurate reports such as the number of cumulative hospitalizations which do not monotonically increase with time, or current hospitalizations higher than cumulative hospitalizations [2], and other inaccuracies such as mortality miscounting [4,5].

2. Estimate Infectiousness Curves

Infectiousness curves, often referred to as generation or shedding curves, represent the amount of virus generated by an individual over time, typically measured as a function of days since infection. These curves are critical for understanding how long and when an infected person is most likely to transmit a virus. They can be derived from direct measurement, estimations, or retrospective calculations. During the COVID-19 pandemic, there was significant uncertainty regarding the infectious period. For example, as late as May 26, 2020, i.e., many months after the start of the pandemic, the Department of Health Services (DHS) was still asking questions about the average infectious period during which individuals can transmit the disease [6]. This suggests that this information was either not widely available or available for very few and did not propagate to the general public or even to modelers engaging in the modeling exercise. One approach to estimating infectiousness is to follow a patient and record the amount of virus they shed as computed in [7]. Each patient may have their unique shredding curve and it may be highly useful to publish the overall distribution of these curves after anonymizing the patient information. In addition to direct measurements, infectiousness curves can also be obtained by retrospective calculations as done in some studies [4,8]. Overall, understanding and accurately estimating infectiousness curves is essential for improving epidemiological models.

3. Estimate Transmission Rate per Encounter

Each encounter between individuals may cause disease transmission, depending upon the disease’s transmission rate. If the probability of transmission per average encounter between two individuals is known, it becomes easier to calculate other related factors. In cases when no prior information is available, this number can be guessed, however it is possible to test the guessed number computationally. However, there are numerous modifiers for the transmission rates that include multiple parameters. For example, transmission rates may differ across regions due to variations in population density, regional climate, and environmental conditions such as ventilation or the presence of public transportation hubs. Such parameters should be enlisted and multiple estimated formulas should be constructed that can be later tested and merged. If information is unavailable, it is possible to make several educated guesses. It is important to note that the transmission rate may be related to the reproduction number (R0), a metric used by epidemiologists to measure infectiousness if the infectiousness, transmission rate, and number of encounters are known. While R0 is a useful higher-level product, particularly for those working with the SIR model, its components may be more comprehensible to less mathematically sophisticated audiences. Therefore, breaking the R0 number into its constituent factors is better for simulation purposes.

4. Estimate Encounter Frequency per Region

Since pandemics are primarily driven by interactions between individuals, it is important to keep track of these encounters. Even though the average number of encounters [9] and its standard deviation [10] are sufficient to start modeling, these numbers can quickly change as people start reacting to the pandemic. Therefore, it is important to develop methods that can estimate the level of encounters based on other factors. Examples from the COVID-19 pandemic include Apple Mobility data [11] that used the number of map searches in Apple devices to estimate how mobile people are, compared to a non-pandemic scenario. Another example is using household size [12] to estimate the number of interactions that an infectious person who is aware of their illness is having after deciding to avoid venturing outside their house. Such household size data is readily available from government census data.

Another important and recent piece of information that can be used to estimate encounter frequency is government decisions on local restrictions. During the COVID-19 pandemic, multiple local jurisdictions applied different rules on school closures, office shutdowns, restaurant restrictions, mask mandates, and mobility limitations. These restrictions were applied on different dates and enforced differently at times with varying degrees of rigor. To the best of our knowledge, there is no dataset regarding the same, neither is this piece of information available at one single location or listed in a centralized fashion. The closest example is the work by Forgoston et al. [13] which specifically targeted the state of New Jersey, US. An extension of the study [14] included all states in an easy format that can be converted to a behavior model. However, multiple versions of that work exist- after all, it is a large undertaking to handle during a stressful time. A similar endeavor was undertaken by Husch Blackwell [15]. However, significant processing is needed to be used in a computational model. A standardized and real-time publication of such data is highly important and therefore should be maintained, or at least supported, by the government. This may be helpful on its own to compare which strategies seem to reduce the effect of a pandemic. These enlisted elements are important to build accurate models that could estimate the number of encounters per day per region. Consequently, these models can be tested against observed data to determine which model best explains the observed numbers.

5. Estimate Mortality Profile

Estimating both the mortality table and mortality time is crucial for accurate disease modeling. The mortality table provides the death proportion of people infected with the disease, ideally stratified by age [16] and/ or other factors like sex, etc. The mortality time provides the time of death since infection. Mortality time and knowing the timing of deaths becomes especially important when one is computationally modeling the disease and wants to validate their models. Also, note that the “time of death since infection” is not the same as the “time of death since symptom onset”. Although it may be possible to provide information on symptom onset since infection, death should be reported as time since infection to support simpler models. Ideally, for modeling purposes, death should be reported as a probability density function that provides the probability of death as a function of “time since infection” and “age”, as demonstrated in a few studies [17]. This approach makes it possible to show it as a table or illustrate it graphically in an interactive manner [18] for less computationally inclined individuals.

6. Estimate the Inaccuracy of Reported Numbers

Reported numbers, even ones that are centralized, accessible, and highly scrutinized, have errors in them partially since data collection procedures are often not consistent, and secondly because there are multiple types of diagnostic tests, each of them having different accuracy and response times. Such errors can create distrust among the general public and even modelers. Therefore, reported numbers have to be audited continuously in various ways. At the onset of the COVID-19 pandemic, initiatives like the COVID-19 Tracking Project attempted to provide “grades of accuracy” to the numbers it gathered and reported per state. Such grades were useful for easily understanding that reported numbers may contain errors. However, few reported numbers are known to be inaccurate when representing an entire population. For example, the number of positive tests in a population does not scale up to represent the number of infections in a population due to factors such as test sampling strategy, test accuracy, and delays. Therefore, it is important to have correction factors to compensate for positive tests and infectiousness in a region. Even reported death numbers, which should be ideally accurate, appear to have errors [4] due to delays in reporting and the cause of death not being associated directly with the pandemic. Therefore, even mortality reports may need correction before entering into models. While modeling, there should be a clear distinction between modeled numbers – which represent virtual infections that should match the ground truth that will be reported in an ideal world, and observed numbers – which represent what we see and report. Modelers should investigate multiple correction strategies to fix the difference between modeled numbers and observed values.

7. Start Simulations that Will Attempt to Match the Recorded Numbers

All collected information should be verified using simulation to see how they match the observed reality. As much as it is tempting to forecast the progression of the pandemic, it is highly unlikely that models can accurately forecast it. Therefore, the focus should not only be on pandemic forecasting but also on explaining and attempting to verify, compare, and construct the assumptions made. Using ensemble modeling tools can help pinpoint errors in one’s comprehension of the disease and help improve it over time. The Reference Model for Disease Progression is an ensemble modeling tool [19] that attempts to gather all models and data, compare and match the numbers, and see what assumptions better fit the observed data that was collected. Note that this should be a continuous validation process so that the understanding improves over time. For example, the first model that was published during the first year of the pandemic [20] estimated the transmission rate per encounter to be 0.6%, while studies conducted during the second year concluded it to be about 2% per encounter. Note that the first publication was performed on data for 2 months starting April 1st, 2020. This study also concluded that a long infectiousness profile is dominant, while more recent publications, which took into account multiple simulation periods of 3 weeks repeating with different starts, have shown that a less contagious infectiousness profile provides better results and that the temperature may have an important role in transmission. Only additional simulations that took much more computing power have shown that both – a highly infectious profile, as well as a less infectious profile, have an influence. Reaching this conclusion required a lot of computing power. Thus, it is useful to repeat the modeling exercise several times on different periods/data to improve one’s understanding. Assumptions/models should be separated from data and the experiments should be repeated for different periods of time to determine which assumptions/models make more sense. Moreover, multiple simulations attempt to extract the invariant disease parameters, i.e., those parameters that do not change due to different behaviors. The more the data is accumulated, the potential for better calculating these parameters as well as understanding the disease increases. However, our ground hypothesis is that the phenomenon can be explained by analyzing the data. Although many modelers believe this is the case, it still needs to be proved in a way that can be perceived as credible by the public at large. Work that systematically checks the compatibility of the assumptions/models against data and reports to as many combinations as possible, can help in establishing this credibility.

8. Make Resources Available and Accessible

All collected information should be made available to the public electronically with as few restrictions as possible. The Creative Commons Zero (CC0) License is recommended while releasing data and codes since it removes all restrictions and makes it as close to the public domain as possible, thus providing incentives for reuse. Even seemingly permissive licenses, such as “Creative Commons Attribution-NonCommercial 4.0 International Public License” that prohibits commercial use or the “GNU General Public License” (GPL) that restricts reuse only under some conditions, should be avoided to allow the reuse of collected information in all sorts of ways. A competition between players should not diminish the incentives for another player, and not hinder its ability to provide useful information that will lead to a better understanding of the pandemic. Moreover, open-source tools and resources should be made readily available to support rapid data aggregation and subsequent modeling efforts. Fortunately, during the COVID-19 pandemic, many organizations made significant strides in providing open-access resources. For example, ad-hoc organizations [2], industry players such as Rescale High-Performance Computinga, Microsoft Cloudb, Amazon AWSc, as well as government-funded academic organizations like the MIDAS Networkd offered valuable tools and resources. However, greater centralization and improved accessibility of data and resources would be beneficial for future pandemics.

9. Improve Measurement of Hospitalization

For accurate hospitalization modeling from the beginning, it’s imperative to capture comprehensive epidemiological data. This includes detailed information on asymptomatic individuals, which requires contact tracing efforts and extensive testing with techniques such as wastewater metric [3]. Without this data, initial models may vastly underestimate the number of cases, leading to inadequate preparation and resource allocation. Ensuring that models account for asymptomatic spread is critical for predicting the true extent of the pandemic and accordingly planning hospital capacities. Another key action is to start real-time simulations that attempt to match the recorded numbers right from the pandemic onset. By continually updating models with incoming data, not only discrepancies can be identified but also corrected promptly. This iterative approach, where models are regularly compared against actual outcomes and adjusted accordingly, helps in refining predictions. Simulations should incorporate various parameters, such as hospital admission rates, length of stay, and ICU needs while adjusting dynamically as new data comes in. Thus, by embedding feedback loops within the modeling process, one can ensure that models remain accurate and relevant, thereby providing reliable guidance for decision-makers. By focusing on these key actions and data points, modelers can significantly improve the accuracy and reliability of hospitalization models from the outset of the next pandemic. This proactive approach will enable better preparedness and response, ultimately leading to more effective management of healthcare resources and better patient outcomes.

10. Include Baseline Models

A recent work [1] systematically analyzed the accuracy of US CDC COVID-19 case forecasting models, by first categorizing them and then calculating their Mean Absolute Percent Error (MAPE), both wave-wise and on the complete timeline of the COVID-19 pandemic in the US. Broadly, the study compared the estimates to government-reported case numbers, one another, as well as two baseline models wherein case counts remained static or followed a simple linear trend. The study highlighted that more than one-third of models fail to outperform a simple static case baseline and two-thirds fail to outperform a simple linear trend forecast.

10.1. For Government Officials

For government officials, the study recommended (i) a reassessment of the role of forecasting models in pandemic modeling and policy formulation, and (ii) raised concerns about “directly” hosting these models on official platforms, like the WHO, US CDC, etc, which are widely trusted by the general public and press. Without strict exclusion criteria, the public may not be aware that there are significant differences in the overall quality of these models. (iii) Current models can be used to predict the peak and decline of waves that have already been initiated and can provide value to decision-makers looking to allocate resources during an outbreak. Prediction of outbreaks beforehand, however, still requires “hands-on” identification of cases, sequencing, and data gathering.

10.2. For Researchers

For Researchers, the study recommended developing pandemic case forecasting models that are able to make accurate forecasts with a focus on increased Horizons i.e., at least 2-4 weeks ahead. To take an extreme example, a model that only forecasts one day in advance would have less utility than the model that forecasts 10 days in advance. Staffing decisions for hospitals can require a lead time of 2-4 weeks to prevent over-reliance on temporary workers, or shortages. They should also focus on data collection methodologies to address the lack of clean, structured, and accurate datasets that affect the performance of forecasting models, which the authors in the present manuscript also attribute as an important area.

11. Expect Variations

Even the best measurements are prone to errors and the best models are imperfect. However, the dynamics of an epidemic involve variations such as mutations that change parameters. Mutations can change infectiousness [21] and mortality and other factors. The spread of mutations may be seen as waves in the pandemic or may mix in the population. Variants such as Delta, the BA variants [22], and Omicron [23] cause additional challenges to compartmental modeling as they have partial “immune escape” so the proportion of the population that is Susceptible and Recovered in a SIR will need to be adjusted accordingly. They also change the R0, or the number of infections caused by the spread of a single infection, which also needs to be adjusted in compartmental modeling [24]. Lastly, because these properties are different by virus variant, machine learning models will become less accurate as they are trained on datasets from previous variants. Quickly adjusting model parameters to new variants is very important and requires accurate data on variant proportions in each population. In many cases, this data was under-reported, restricted, and difficult to obtain. It also was on average 1 month behind in reporting numbers by labs. Having accurate, timely, and accessible data about new variants is essential to modeling. Potential immune escape by variant also needs to be communicated to the public, as it may mean that previously infected or vaccinated individuals are no longer immune to infection and should take appropriate precautions.

12. Determine Means of Disseminating Information to the Public

Beyond creating physical portals for public access, it is also important to make the public aware of the limitations of models and the data collected. This can prevent people from overreacting based on the forecasts of a specific model in mind. It is important to educate the public about what can go wrong with the data and models and always be cautious with interpreting numbers. If a certain model suggests strict measures to be taken, it’s important to explain that the strictness is solely based on that model, and it is wise to show what other models suggest. Presenting a larger picture can help mitigate both public anxiety and indifference, which sometimes relies on guidance from the media or authorities. Moreover, when reporting model results it is important to trace back all information regarding the model and its data sources to ensure reproducibility before reporting.

Providing accurate information to the public at large is important to engender their trust and to help ensure they follow enacted policies. Early on in the pandemic, trusted authorities have made errors in order to affect behaviors. This was short-sighted and led to mistrust by the public later and created behavior changes. We are trying to focus on modeling in this manuscript and recognize that behavior is really difficult to model and has an effect on our modeling efforts. Therefore we call authorities worldwide to admit errors quickly and humbly. This may have a positive effect aside from modeling efforts. The actions described above require adequate infrastructure for efficient execution. Below are some key elements that will accelerate analysis:

12.1. Computing Infrastructure

Long-range forecasting as well as simulations require high-performance computing power. Fortunately, cloud computing is now readily available on demand today, with industry and government funding to access it. Nevertheless, software tools need to be modernized to effectively leverage these platforms. To give an example, this manuscript was written based on the experience earned while modeling an ensemble model [25] during the COVID-19 pandemic. The simulation “engine” that runs The MIcro Simulation Tool (MIST) [26] was imagined almost two decades ago when cloud computing had hardly started to appear. Newer simulation engines like covasim [27,28] and Vivarium [29,30] now exist. However, to date, none of them have been demonstrated to support an ensemble that can “explain” the pandemic. The Reference Model is currently the only ensemble shown to be capable of this protected by two US patents [31,32]. Thus, there is a need to boost the computing infrastructure, i.e., not only the hardware but also the software tools and its governing legal infrastructure.

12.2. Centralized Epidemiological and Clinical Records

A few centralized data infrastructures dedicated to shared databases or data lakes with public access would allow to storage of structured and unstructured data at any scale and enable organizations to perform different types of analytics, from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions. These infrastructures would allow scientists from all over the world to benefit from the collected information seamlessly, without struggling with data format differences, incomplete records, or inconsistent imputation methods.

We have seen examples of well-organized datasets with dashboards that visualize epidemiological data, such as cases, deaths, mortality risks, and vaccinations. Notable examples included epidemic indices on platforms like COVID-19 Observatorye, Our World in Data [33], and antigenic variances on Nextstrain [34]f and GISAIDg. However, what remains lacking is comprehensive clinical data, including detailed information about hospitalized patients, their health conditions, examinations, and other pertinent details. Essentially, the electronic health record of infected individuals contains invaluable scientific information on the disease course, particularly when related to the concept of digital twins. For instance, during COVID-19, the mathematical modeling community found viral load, antibody titers, and cytokine concentrations to be some of the most crucial data for developing models of the immune response to the pathogen. However, this critical information did not become available through scientific publications until 3-4 months after the pandemic began.

Setting up such uniform infrastructure faces several significant obstacles and needs careful consideration and balancing. Organizationally, different institutions and countries may have varying standards and protocols for data collection and storage, making integration challenging. Politically, there can be issues related to data sovereignty, where governments may be reluctant to share data freely due to concerns about privacy, security, or geopolitical advantages. Maintaining a centralized system requires robust governance frameworks and substantial coordination among diverse stakeholders as well as close legal scrutiny. A common agreement on the way to share and access information is crucial yet may be impractical. Without this, the full potential of centralized data infrastructures cannot be realized, and the global scientific community will continue to face barriers in their efforts to combat epidemics effectively. Nevertheless, the system has to make an effort to reach the best centralization possible within the limitations. Attempts have been made during the pandemic to centralize important data such as the MIDAS network attempt to centralize COVID papersh and the effort by DHS to summarize what is known about the pandemic [6]. However, those attempts focused on epidemiological data rather than clinical data that is more restricted. If clinical data is accessible in some way, that would considerably help model development.

12.3. Educate and Train Experts on Recently Emerging Technologies

The disease modeling community still relies on almost a century-old SIR model [35] as a base for many models. Newly emerging ideas like using an Ensemble of Models (popular in cross-domain fields like Machine Learning) are generally uncommon in the disease modeling community. Therefore, very few of them were created during the COVID-19 pandemic. Researchers should be educated on emerging technologies that will be heavily used in the next pandemic. Moreover, since many experts in disease modeling do not have an in-depth programming background, they should be trained in the use of existing computing tools such as version control, database systems, etc., that can help in the rapid and reliable exchange of information that is both traceable and reproducible. Moreover, organizing modeling drills, and Hackathons for modelers and researchers based on synthetic data, wherein both national and local governments also collaborate to learn their roles in data collection and reporting, can help create a system of trained researchers and government personnel who will be better prepared for the next pandemic.

12.4. Educate the Press Before Educating the Public

Beyond educating experts who are directly involved in the modeling process, an effort should be made to educate the general press, and subsequently the public about the capabilities as well as shortcomings of these models. This can be achieved through various channels, such as simple websites, articles on webpages that have high traffic, or educational TV programs. Though it is difficult to educate everyone, there should be more emphasis on educating the press that disseminates the information to large audiences. Broadcasting brief media interviews with modeling experts can help people understand how these models work, their limitations, and their accuracy. Presenting this information as simple conversations, without using jargon, will make it more accessible and helpful. Journalists tend to have fluent conversation and articulation skills compared to researchers and therefore may be able to convey the same to the public in an easy and appealing way. However, journalists should avoid glorifying models and particularly prevent the researchers from doing so (especially glorifying that their model is the best compared to others)– since the larger audiences might be misled and develop a bias for a particular model without realizing that a model’s forecasting accuracy tend to vary over time due to changes in external environments.

12.5. Research

Micro-simulation or agent-based simulations have been widely explored. However, implementing them for High-Performance Computing (HPC) infrastructures still needs more research. For example, new hardware combinations such as GPUs alongside recent compiler technology like Numba [36,37] or CUDA parallelization [38] need more exploration. Most modelers focus on improving simulation speed; however, an equally important area is to improve the assembling speed of ensemble models. For example, one ensemble model this paper is based on [19,39] is a prototype that started with the pandemic based on existing technology to model diabetes. To date, it took roughly 4 years – with over 70 model versions, and hundreds of CPU years of computing to reach conclusions. Early publications of the model were different, partially due to insufficient data. Yet, some necessary modeling techniques have appeared during the last years of development that improve understanding and change conclusions. One specific example is the need to repeat simulations with different start times of the cohort that changed the model conclusions. More research into this technology is needed as well as more tests. After the COVID-19 pandemic, modelers have more datasets to work with and can effectively study how to improve the data collection, modeling, and reporting cycle to be able to provide a more accurate view of the disease in a reasonable time in the next pandemic. It is still an open question of how fast one can come up with a good explanation of a disease since its outbreak. Another unanswered question is if we can and under what conditions/ constraints can apply this research to a future pandemic. More experimentation is needed to help answer such open questions.