Global Comparison of Changes in the Number of Test-Positive Cases and Deaths by Coronavirus Infection (COVID-19) in the World

Comment: Comment:
There are a few fatal flaws in this study.
First of all, you are confusing correlation with causation.
Secondly, there is no statistical control and verification of the target population.
Third, the results of machine learning, which is a classifier, are being misused without any verification.
The authors claim that, "You there! One lemon contains as much vitamin C as one lemon! ", is merely saying.
Each point will be addressed bellow.

Response:
Thank you for your comments on our paper. There are two major findings in this study. One is the large global and regional differences in COVID-19 epidemics. Surprisingly the numbers of confirmed cases and deaths per unit of population differed approximately 100-fold between Western and Asian countries. The other is the relationship between rates of deaths and the positive PCR results in Western countries where the number of countries is relatively large. I understand that all the issues you pointed out are focused on the second finding, especially the soundness of our hypothesis and the statistical method used for the analysis. Since this study is an ecological one a causal relationship cannot be proven as you pointed out. It is natural that many factors are involved in the spread of infection in each country, and since reproducible experiments are not possible, a causal relationship cannot be proved strictly for any infection spread. In that respect, we agree that some of our descriptions were excessive and should be corrected. We should carefully examine the factors that found the correlation and the possibility of confounding.
We couldn’t establish a complete control group in this study simply because the situations in the countries differed so much. It is obvious that the analysis using full epidemiology information is ideal and should be done for preparation for the future pandemic of another emerging or re-emerging infectious disease. However such analysis can be made after the epidemic becomes controlled in many of the countries and data become available. We performed this study considering the current tragic situation of COVID-19 in the world, which needs various urgent decisions based on the available data at present. Machine learning was used to predict the progress of the infection in a relatively short period for each country group because the progress of the infection was not sufficient in some countries at the time of observation. Machine learning was not used as a classifier. We submitted the source code of analysis to the journal but it was not attached to the preprint. Therefore, we should describe it clearer in the manuscript. We thought that a comprehensive analysis to determine various affecting factors of the infection was yet difficult given that countries have so many attributes and that the progress of the infection was not sufficient in many countries.

Comment: 1. misunderstanding of correlation and causation
The author states that there is no relationship between the number of tests and the number of deaths adjusted for, but argues that there is a correlation between the "positive rate" using the number of tests as the denominator and the "number of deaths". In other words, it is obvious that there is a correlation between the number of confirmed infections and the number of deaths in the numerator of the positive rate. However, the authors purport to be the first to find a correlation between the positive rate and the number of deaths, and the logical development is already broken and fatal at this point.

Response 1:
Regarding this point, it is necessary to consider differences between an increase in the number of test-positive cases (NPDP) and an increase in deaths (NPDD), and it is true that sufficient explanation was not provided in the original paper, so we need to add it. As shown in the supplemental figure (added in version 2, This version was not changed other than an addition of this figure), there was no difference in the degree of increase in NPDP up to two weeks after BPI in countries with low and high positive rates. Your point is that since the positive rate and the number of test-positive cases were directly related, it is very natural that the positive rate and the number of deaths were likely to be related. However, in the early stage of the epidemic, changes in the number of test-positive cases were independent of the positive rate.
From the point of view of your comments, the positive rate of countries with a low positive rate had become low as a result of the smaller infection spread. Therefore, the difference between countries with a low positive rate and a high positive rate would be small before the infection spread. In this case, it may be natural that differences in NPDP were small between these countries in the early stages of infection. Or, from a conservative perspective, if countries with a low positive rate tended to perform a large number of tests from the beginning, it is possible that more positive cases were found. In this case, even if the number of test-positive cases was seemingly similar, the extent of infection spread would be low in countries with a low positive rate, and also the number of seriously infected people would be actually lower than in countries with a high positive rate.
However, in both cases, it cannot be explained that the slope of change in NPDP was attenuated only in countries with a low positive rate at two weeks after BPI. We think this attenuation of the slope at two weeks after BPI is very important and is actually the implicit target of our analysis. The slope after this attenuation was reflected over time in the slow increase in NPDD in countries with a low positive rate shown in Figure 5B in the paper. The change in the slope of NPDD would have caused the difference in the lag time between BPI and BDI which was also discussed in the paper. Therefore, the change in NPDD 30 days after BDI can be regarded as a reflection of the change in the slope of NPDP. Given this background, we thought that it is reasonable to compare NPDDs in countries with a different positive rate for assessing a depressing factor that influences the rate of spread of infection. In addition, considering that approximately two weeks are necessary for the detection of test-positive cases after the infection, the policy of testing in each country can be a possible causative factor that impacted the change of the spread of epidemic from the detection of test-positive cases.
As stated in the text, the absolute value of NPDP depends on the testing policy by each country (but please note that relative changes of NPDP in each country are yet informative). Therefore, NPDD, which is not affected by such differences, was considered more direct than discussing the change in NPDP. In addition, it can be only confusing to discuss differences in BPI and BDI when changes in NPDP is considered. On the other hand, when discussing the basis of causal relationships as you pointed out, the use of NPDD needs detailed explanations for specialists as provided here. I would like to consider which of these issues should be discussed, based on the reviewers' points of view. Your comment provided us a good opportunity to explain deeper into this issue. I thank you again.
An additional point to note is that if it is true that sufficient tests are effective in controlling the spread of infection, the positive rate is a good index of the sufficiency of the test and the number of population-adjusted deaths is a good index of the infection spread. Thus, it seems theoretically impossible to completely solve the causal problem which was pointed out by you in this case. Even if it is true, it cannot be proved eternally. Therefore, it is an important option for understanding the current situation to make comparisons as done in this study with careful considerations of causal relationships and confounding issues.
Successful isolation of infected cases from the city in countries that were well-tested from the outbreak of infection may result in the subsequent spread of infection being suppressed as suggested by the slope of NPDP. It seems reasonable to assume that the decreased seriously infected cases reduced the overburden on the medical system and provided sufficient medical treatment, leading to a significant reduction in deaths. As you pointed out, these are only an interpretation and we also need to describe that, in countries where the positive rate was low, the possibility that confounding factors such as high medical capacity or different national policies or characteristics, age composition, and ethnic composition may also be involved in this result. In addition, we think that 7% of a positive rate may be only a snapshot of various changes during the epidemic. The positive rate of the PCR test is informative but the absolute target value is difficult to determine.

Comment: 2. statistical controls are not in place and a lot of bias remains
The authors introduced an observational variable, the positivity rate, which does not go up or down by a single variable, the number of tests, but depends on the number of confirmed infections of the molecules tested. The actual status of the infection and the probability of positive results due to the testing method are still uncertain at present, and selection bias is very high. Furthermore, the authors deliberately exclude data from countries that are disadvantaged in deriving their results.
If the positive test rate is to be used as an observational variable, these baseline conditions should be statistically controlled to be within a certain range across countries, or they should be shown to be already within a certain range. However, there is no indication that the authors exercised any control over the inspection situation. It is a misreading because it does not take into account the vastly different testing regimes, hospitalization conditions, and timing of treatment interventions in each country into account at all, and it begins to develop its logic using the positive test rate figures as a goal.

Response 2:
As far as we know, many countries are using the PCR test as the basis for final judgment, and even when other methods such as the antibody test or image diagnosis are used together, there is no major conflict with the PCR test result. Therefore, I do not think that the selection bias due to the difference in the inspection method is very high for COVID-19. Since this study targeted the fatalities, we had no choice but to limit the countries subject to analysis to those with a certain degree of spread of infection. The selection of the country this time is based on an objective indicator that the number of infected persons is above a certain level, and there is no fact that a specific country was not intentionally analyzed. Of course, infections are progressing rapidly and the number of countries that can be analyzed is increasing at present.
There is no baseline value before the infection because the positive rate is the number that appears as responses to the infection. Therefore such control is impossible. I actually think that the change in the positive rate during the spread of infection should also be included in the complete analysis in the future. But at present, detailed information is not available in many countries in the world. I would like to make it when the information is obtained.

Comment: 3. given the relationship between the low positive rate and the low number of deaths in the data, isn't the correct interpretation in light of "Occam's Razor" to assume that the number of deaths is low because the infection is not widespread?

If we use the Directed acyclic graph (DAG) to organize the causal flow, we can see that the story of this paper is fatally wrong. The authors' logic, if presented in DAG, is as follows.

"More tests" → "lower positive rate" → "fewer deaths"

The authors' logic is that increased positive rates reduce deaths. But there is no causal relationship between the two numbers of positive rates and the number of deaths." It is reasonable to assume that the common cause of both the "positive rate" and the "number of deaths" is the "degree of spread of infection" in the first place, and that the two phenomena of "positive test" and "patient death" arise from this.

"Spread of infection," "more positive tests."
↓arrow
"The number of deaths will increase."

Response 3:
I think our response to comment 1 explains the answer to this comment.
Our logic is like this.
"More test" -> "effective isolation" -> "reduce epidemic" and "lower positive rate"
->" fewer death" Of course, this is one possible interpretation of the current observation.

Comment: 4. the authors have not verified the validity of the classification as to why the result is 7%, which is a typical example of misuse of statistics and machine learning.

The authors write that they used a gradient boosting decision tree. Machine learning returns some relationship as a result, with or without validity. We find "structures" and "features" from vast amounts of data that cannot be found in human cognition, but they are not causal relationships. The gradient boosting decision tree is a classifier and does not establish a causal relationship at all. I'm just categorizing the pattern of deaths and positive rates. The author has only categorized them into three groups, and there is no explanation for the cause there. Also, there is no statistical validation of anything. You need to interpret the fit with other validation data, not the training data used directly for the analysis. However, there is no indication of any such verification.

Response 4:
As mentioned at the beginning, the use of machine learning in this study is not for identifying the influencing factors, but to predict the number of deaths as a group because the spread of infection was still insufficient to assess the number at 30 days after BDI in many countries. The reliability of the prediction is evaluated by the bootstrap method. It was judged that there is a clear difference in the estimated values as a group because the confidence interval does not intersect at all. The thresholds of 7% and 17% were adopted as values close to the quartile. We will mention these explanations as well in the paper.
Given the myriad of attributes in a country, it is not possible without a hypothesis to find the factors that contribute to the observed reduction in deaths. Thus, it is necessary to select a rational hypothesis and consider its superiority or inferiority, but since it constantly fluctuates depending on the spread of infection, it is unavoidable to make the best of the current available information.
I found a sentence in the Discussion that "machine learning analysis classified ..." and this was misleading. It should be corrected and at the same time, we would like to add an explanation in the appropriate place to clarify why we used machine learning.

Comment: We added the supplemental Figure 1 to version 1, not to version 2. This was in accord with a suggestion by Preprints.