A “Clinician’s Probability Calculator” to convert pre-test to post-test probability of COVID-19 , based on method validation from each laboratory

Background During coronavirus pandemic testing and identifying the virus has been a unique and constant challenge for the scientific community. In this paper, we discuss a practical solution to help guide clinicians and public health staff with the interpretation of the probability that a positive, or negative, COVID-19 test result indicates an infected person, based on their clinical estimate of pre-test probability of infection. Methods The authors postulated that the clinical pretest probability of COVID-19 increases relative to local prevalence of disease plus patient age, known contact, and severity of symptoms. We conducted a small survey on LinkedIn to confirm that hypothesis. We examined results of PPA (Positive Percent Agreement, sensitivity) and NPA (Negative Percent Agreement, specificity) from 73 individual laboratory experiments for molecular tests for SARS-CoV-2as reported to the FIND database,(1) and for selected methods in FDA EUA submissions (2,3). We calculated likelihood ratios to convert pre-test to post-test probability of disease, then further calculated the 2 number of true and false results expected in every ten positive or negative test results, plus an estimate that one in ‘x’ test results is true. We designed an online calculator to create graphics and text to fulfill the objective. The LinkedIn survey confirmed that the pre-test probability of COVID-19 increases with patient age, known contact, and severity of symptoms, as well as prevalence of disease in the local population. PPA (Positive Percent Agreement, PPA) and NPA (Negative Percent Agreement, specificity), differ between individual methods. Results vary between laboratories and the manufacturer for the same method. The confidence intervals of results vary with the number of samples tested, often adding a large range of possibilities to the reported test result. The online calculator met the objective. A positive or negative test result from one laboratory conveys a higher probability for the presence or absence of COVID-19 than the same result from another laboratory, depending on clinical pre-test probability of disease plus proven method PPA and NPA in each laboratory. Likelihood ratios and confidence intervals provide valuable information but are seldom used in clinical settings. We recommend that testing laboratories verify PPA and NPA, and utilize a tool such as the “Clinician’s Probability Calculator” to verify acceptable test performance and create reports to help guide clinicians and public health staff with estimation of post-test probability of COVID-19 .

the probability of true and false results has never been so critical. This article explores a modified application of likelihood ratios to provide practical guidance to the relative probability of true and false results.

What are pre-test and post-test probability?
Pre-test probability and post-test probability are the probabilities of the presence of a disease (such as COVID-19 ) before (pre) and after (post) a diagnostic test. (6) In some scenarios a diagnostic test may not be of help and may lead to increased confusion especially when the pre-test probability of a disease is either very high or very low. With COVID-19 and the importance of pre-symptomatic or asymptomatic patients however, tests are often performed even with low pre-test probability Although sometimes confused with simple prevalence of disease (7), the clinical pre-test probability of disease can be more precisely estimated with clinical information on each patient. In a small survey with 16 respondents on LinkedIn, the authors asked healthcare professionals to "estimate the probability that a 20-to-30-year-old patient, and 60-to-70-year-old patient, actually has, or will soon develop COVID-19 infection?" Symptoms ranged from i) none, to ii) sore throat, and nasal stuffiness, to iii) sore throat, and nasal stuffiness, with reduced taste or smell to iv) sore throat, and nasal stuffiness, with reduced taste or smell, fever, and body ache. We found, as logic and clinical experience would dictate, that pretest probability increases with local COVID-19 prevalence, patient age, SARS-COV-2 exposure history and clinical symptoms.
Post-test probability is driven by pre-test probability and the likelihood ratio of the test methodas that method was verified in each laboratory. The likelihood ratio is driven by test PPA and NPA. PPA and NPA vary between methods as reported by manufacturers to FDA for EUA evaluation, and between laboratories using the same method. Different labs choose different numbers and criteria for known positive and known negative samples.
Clinicians and public health professionals who interpret test results are not always provided with the Fact Sheet for Healthcare Providers provided by test manufacturers that describe the interpretation of positive and negative test results. Different tests are approved for different clinical situations; few are approved for asymptomatic patients. The following is an excerpt from the "Fact Sheet for Healthcare Providers" for both Manufacturers 1 (8) and 2 (9): Manufacturer 1 (8): "This test is to be performed only using respiratory specimens collected from individuals who are suspected of COVID-19 by their healthcare provider within the first seven days of the onset of symptoms." Manufacturer 2 (9): "This test is to be performed only using respiratory specimens collected from individuals suspected of COVID-19 by their healthcare provider." Both manufacturers: "What does it mean if the specimen tests positive for the virus that causes COVID-19?
A positive test result for COVID-19 indicates that RNA from SARS-CoV-2was detected, and therefore the patient is infected with the virus and presumed to be contagious. Laboratory test results should always be considered in the context of clinical observations and epidemiological data (such as local prevalence rates and current outbreak/epicenter locations) in making a final diagnosis and patient management decisions.
The SARS-CoV-2test has been designed to minimize the likelihood of false positive test results. However, it is still possible that this test can give a false positive result, even when used in locations where the prevalence is below 5%.
What does it mean if the specimen tests negative for the virus that causes COVID-19?
A negative test result for this test means that SARSCoV-2 RNA was not present in the specimen above the limit of detection. However, a negative result does not rule out COVID-19 and should not be used as the sole basis for treatment or patient management decisions. It is possible to test a person too early or too late during COVID-19 infection to make an accurate diagnosis via the SARS-CoV-2test."

What are likelihood ratios?
The likelihood ratio is a tool used in evidence-based medicine to assess the value of performing a diagnostic test. It uses PPA and NPA to create a ratio of the probability that a test result is correct to the probability that it is not. A likelihood ratio is the percentage of ill people with a given test result divided by the percentage of well individuals with the same result (true result: false result). Ideally, abnormal test results should be much more typical in ill individuals than in those who are well (high likelihood ratio) and normal test results should be more frequent in well people than in sick people (low likelihood ratio). Likelihood ratios near one have little effect on decision-making; by contrast, high or low ratios can greatly shift the clinician's estimate of the probability of disease. When combined with an accurate clinical diagnosis, likelihood ratios improve diagnostic accuracy in a synergistic manner. (10,11) Tests can be either positive or negative, so there are two ratios: • Positive LR (LR+): This tells us how much to increase the probability of having a disease, given a positive test result. The ratio is: "Probability a person with the condition tests positive (a true positive) / probability a person without the condition tests positive (a false positive)." (10) • Negative LR (LR-): This tells us how much to decrease the probability of having a disease, given a negative test result. The ratio is: "Probability a person with the condition tests negative (a false negative) / probability a person without the condition tests negative (a true negative)." (10,11) Likelihood ratios are calculated to determine 2 things: i) how useful a diagnostic test is and ii) how likely it is that a patient has a disease. (10) Likelihood ratios range from zero to infinity (9999.9). The higher the value, the more likely the test will indicate that the patient has the condition.
Likelihood ratios are calculated from PPA and NPA:

What are Confidence Intervals?
"A confidence interval gives an estimated range of values which is likely to include an unknown population parameter." (10,11) Confidence intervals provide a range of possible results: minimum, probable and maximum. They tell the end-user how much faith they can have in the value reported. Methodology 1. We created a LinkedIn survey asking for "your estimate that a 20-to-30-year-old patient (or a 60-to-70-yearold patient) actually has, or will soon develop COVID-19 infection" -with local prevalence of 3%, with and without known contact, escalating age and COVID-19 symptoms.
2. We used the calculations and definitions in Table 1 to examine results of individual laboratory experiments for molecular tests for SARS-CoV-2as reported to the FIND database (1), and for selected methods in FDA EUA submissions (2,3).  Results:

LinkedIn Survey:
Seventeen people responded to a LinkedIn survey asking for "your estimate that a 20-to-30-year-old patient (or a 60-to-70-year-old patient) actually has, or will soon develop Covid19 infection" -with escalating Covid19 symptoms, with and without KNOWN contact." The local prevalence was given as 3%. (Questionnaire attached as supplementary material). Table 2 presents the survey results, showing a clear pattern that pre-test probability increases with patient age, known contact and presence of typical COVID-19 symptoms.  The authors observed no relationship between the number of samples tested and method quality reflected in the reported PPA and NPA.      When pre-test probability raises to 50%, (rows 12-21) post-test probability rises to over 90% for these labs.
Even with a negative test, post-test probability is approximately 20% in Labs A and B (row 15). In row 18, with pre-test probability of 50%, only 8 of 10 negative tests are true for Labs A and B.    Where Table 4 presented the 'Probable' Post-Test Interpretation of Results, Table 6 shows the range of possibilities with low and high confidence intervals. Notice in row 4 that, where Mfg-1 and Lab A both reported 100% PPA, confidence intervals show that could actually be as low as 18% or 7.5%; in Lab C, the reported 100% may actually be as low as 2.8% due to the low number of samples tested. Row 9 shows that there may be less than two or three true results in every ten positives reviewed by clinicians.  Tables 3, 4, 5 and 6.
The calculator report clearly conveys information to interpret results, based on the test used and verified in each laboratory. In the graphic, the x-axis is the pre-test probability, as estimated by the clinician or public health   can be converted to likelihood ratios which can be used to convert clinical pre-test probability of disease for a specific patient to post-test probability.

Relevance of likelihood ratios:
Likelihood ratios allow one to convert pre-test to post-test odds of infection. The mathematics of this process are complicated, but the logic is clear. (15) When pre-test probability is 50%, the odds are 1:1 that the patient is infected. One of every two people with 'these' clinical symptoms is expected to be positive before testing (50%). If the positive likelihood ratio is approximately 24, as in Lab D in Table 3, multiplying the pre-test odds

Importance of number of known samples tested:
The number of known positive and negative samples tested determines the confidence intervals around PPA and NPA (14). Figure 1 and Table 3 Table   5.) The low number of known samples tested in Lab C do not allow this lab to verify acceptable method performance.

Impact of confidence intervals:
Confidence intervals determine the range of possibilities for PPA and NPA, which drive likelihood ratios that drive post-test probability of COVID-19 with positive and negative test results. Post-test probability drives the number of true and false positive and negative tests in every 10 positive or negative results seen, and how many positive or negative test results would be seen to find one true test result. Confidence intervals allow users to visualize the gap between the post-test probability that a positive, or negative, test indicates an infected person.

Value of graphs and metrics reported by the Probability Calculator:
Instead of either taking all positive or negative test results at face value or developing personal experience to 'guess' if results are true or false, clinicians can visualize a reliable scientific range of possibilities. Glancing at the six graphs in Figure 5 clarifies that when pre-test probability is only 3%, the probability of a person having COVID-19, even with a positive test, is less than 50% -except in Lab B where they tested enough samples to prove test reliability. These graphics and data eliminate the use of Fagan's Nomogram (7), which is typically used with likelihood ratios but are cumbersome for front-line use and does not include confidence intervals.
Laboratory directors and public health officials who are challenged to select and verify test methods can clearly see the ability of each test to project, or rule out, COVID-19 infection. Lab C shows no gap at all between the range of possibilities that a positive or negative test indicates an infected person. The test has not been verified to provide useful information by testing only five known-positive and three known-negative samples.
Laboratory directors and clinicians can have little confidence in the values reported.
Clinicians can benefit from understanding the number of true and false positive results they can expect to see in every ten positive or negative results. Clinicians may see as few as one or two true positives in every ten positive test results according to the Manufacturer-1 and Lab A, while Lab B can be relied on to produce over seven of ten true positives (