Evaluating Masks Through the Lens of Evidence-based Medicine

1. The Mask Issue

1.1. RCTs on Masks

The effectiveness of face masks in curbing viral transmission has been tested in several randomized control trials before and after the onset of the COVID-19 pandemic. Two large RCTs on COVID - the DANMASK [6] and the Bangladeshi [7] trials, have been conducted and published. The DANMASK RCT, carried out between April and May 2020 in Denmark, involved the largest set of participants (almost 5000) in a mask RCT up to that time. The trial results showed a non-significant reduction in the odds ratio of the masked group relative to the non-masked control arm (0.82 [ 0.54 - 1.23]; p = 0.33). The study conclusion was that “the recommendation to wear surgical masks to supplement other public health measures did not reduce the SARS-CoV-2 infection rate among wearers by more than 50% in a community with modest infection rates, some degree of social distancing, and uncommon general mask use.” The reference to a 50% reduction was because the trial was powered to find at least this degree of risk reduction. Despite the fact that the RCT involved a large number of participants and was well-conducted as possible by a pro-mask advocate principal author [8] the mainstream medical-scientific community did not spare efforts to diminish the study impact on the ineffectiveness of masks [9,10,11,12].

The cluster RCT with the largest number of participants in this type of trial (300,000) was carried out in 600 villages in a Bangladesh rural zone. Both cloth and surgical masks were tested, the former showed no reduction in COVID transmission while surgical masks provided a 9.5% reduction in symptomatic SARS-CoV-2 seroprevalence and an 11.6% reduction in the prevalence of COVID-like symptoms (both values significant at the 5% level). In total, there were 1086 cases of symptomatic seropositive infection in the control arm and 1106 in the treatment arm. This small reduction was hailed by some in the mainstream medical-scientific community as a definitive proof of masks’ effectiveness [13,14,15]. However, even this small effect is disputed, since a reanalysis of this RCT found that unblinded staff may have contributed to significant bias in how they approached the treatment and control groups [16]. A surprising finding in this study was that masking effectiveness differed considerably depending on the age of the wearer. While the 50-59 and ≥ 60 years old groups displayed significant reductions in COVID-19 seroprevalence (0.772 [0.595-0.949] and 0.647 [0.448-0.845], respectively), the younger age groups, (< 40 and 40-49 years old), did not show any significant reduction in seroprevalence (0.967 [0.834-1.100] and 1.009 [0.817-1.200], respectively). The authors of the paper provided some hypotheses to explain this discrepancy, but none have been tested. More recently, a randomized trial comparing the effect of medical masks and N95 respirators to prevent COVID-19 in 29 health care facilities in Canada, Israel, Pakistan, and Egypt has been published [17]. No significant difference between the N95 and medical masks arms in the intention to treat analysis was observed. RT-PCR–confirmed COVID-19 occurred in 52 of 497 (10.46%) participants in the medical mask group versus 47 of 507 (9.27%) in the N95 respirator group. The pooled hazard ratio was 1.14 [0.77 - 1.69]). This was the largest RCT on N95 respirators ever performed.

1.2. The Cochrane Review on Mask RCTs

All randomized trials conducted before the COVID pandemic have reported null to low effect of masks and no difference between masks and respirators. The best RCTs of this kind were systematically reviewed by Cochrane researchers. The fifth and most recent review of this kind was published in January 2023 [18]. Cochrane is an international network whose mission is to analyze and summarize the best evidence from biomedical research providing authoritative and reliable information, unconstrained by commercial and financial interests, being the leading global advocate for evidence-informed health and care (https://www.cochrane.org/). Cochrane reviews are internationally recognized as the benchmark for high-quality information about the effectiveness of health care. Over 7,500 reviews have hitherto been published, they are freely available from the Cochrane library (http://www.cochranelibrary.com/). In the 2023 review on masks and other interventions the authors analyzed 13 RCTs on medical or surgical masks, of which 11 were in the community and 2 in healthcare workers. In addition, five studies on N95/P2 respirators (4 in hospital or outpatient settings and one in the community) were also evaluated. All of these trials studied the spread of influenza or influenza-like illnesses (ILI). Medical or surgical masks were compared to no-masks and respirators were compared to medical or surgical masks. The most reliable outcomes were from studies that used laboratory-confirmed viral illness, which provided a “moderate certainty of evidence”. These studies showed no significant risk reduction by medical/surgical mask or P2/N95 wearers. The total number of participants in the RCTs on medical/surgical masks versus no mask was 290,000, with approximately half this number in each arm. The pooled risk ratio (RR) for laboratory-confirmed influenza was 1.01 [0.72, 1.42], while the pooled RR for influenza-like illness was 0.95 [0.84, 1.09]. A pooled RR of 1.10 [0.90-1.34] was observed when N95 respirators were compared with medical/surgical masks in laboratory-confirmed influenza, a result that favors surgical/medical masks, though not statistically significant. Interestingly, the two other viral illness outcomes – clinical respiratory illness and influenza-like illness, which provided less certainty of evidence, also showed statistically non-significant results, but with pooled risk ratios favoring N95 masks (0.70 [0.45, 1.10] and 0.82 [0.66, 1.03], respectively). In general, there is an inverse correlation between mask study quality (or quality of evidence) and mask effectiveness, i.e., the better the study or the quality of evidence, the lower the effectiveness of masks. This becomes evident while comparing more rigorous and less rigorous studies (randomized vs. observational for example) or different endpoints – clinical respiratory and influenza-like illnesses (less reliable endpoint) vs. laboratory-confirmed influenza (more reliable endpoint) in a single study as in the case of N95 vs. surgical/medical masks mentioned above. The Cochrane review on mask RCTs [18] concluded that “The pooled results of RCTs did not show a clear reduction in respiratory viral infection with the use of medical/surgical masks. There were no clear differences between the use of medical/surgical masks compared with N95/P2 respirators in healthcare workers when used in routine care to reduce respiratory viral infection”. There are no qualitative differences between the current and previous Cochrane review [19]. If anything, the effectiveness of masks decreased with the addition of three recent randomized trials to the 2023 review studies [6,7,20], two of them on SARS-CoV-2 transmission. It should be noted that all but one N95 trial, including the most recent international multicenter RCT [17] were conducted with healthcare workers who had been trained on how to fit and wear respirators. Considering the failure of these trials to demonstrate a reduction in viral transmission compared to surgical/medical masks, it makes one wonder if the recommendations for wearing respirators and the calls for their distribution to the general public (https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/types-of-masks.html, https://www.ama-assn.org/delivering-care/public-health/what-doctors-wish-patients-knew-about-wearing-n95-masks, https://edition.cnn.com/2022/01/19/politics/n95-masks-biden-administration-covid-19/index.html), who are not trained to wear these type of masks, will be of any benefit.

1.3. Meta-Analyses that Support Masking

Several reviews and meta-analyses have evaluated the data from mask studies since the beginning of the COVID pandemic and concluded that mask wearing may be associated with a significant reduction in viral transmission [21,22,23,24,25,26,27,28]. We will look at two of these reviews in more detail. A meta-analysis published at the same time as the Cochrane review - December 2020 - examined 12 randomized trials and 21 observational studies on the effectiveness of masks in preventing the transmission of respiratory viruses [29]. According to this review, face masks reduced the risk of primary respiratory infections by 6 - 15%. The authors justified the inclusion of observational studies because "the conclusions of RCTs and observational studies differ widely and both are prone to significant bias". However, since randomized trials are designed to minimize bias and confounders, their inherent bias is considerably lower than that of observational studies [2]. Additionally, the risk reduction calculated in this meta-analysis (6-15%) for mask wearers is too low to justify a mask recommendation, let alone mandates (see section below “on masks and condoms” for a comparison of mask and condom effectiveness). An exception to the pro-masking chorus of meta-analyses that reviewed observational studies is Wang et al. who, by analyzing 5 randomized and 10 observational studies estimated a non-significant pooled odds ratio of 0.96 [0.8–1.15] for surgical mask effectiveness against acute respiratory illness among individuals in non-healthcare settings [30], concluding that surgical mask wearing among individuals in the community is not significantly associated with a reduction in acute respiratory illnesses. Meta-analyses that concluded that masks do have a substantial impact on viral transmission relied only or almost exclusively on observational studies. This is exactly what the very influential paper by Chu et al. has done [21]. The pooled effect of 29 observational trials (26 in healthcare settings and 3 in the community) and not a single RCT reached a significant adjusted odds ratio of 0.15 [0.07 – 0.34]. There were other recent meta-analyses on masks that included only or mostly observational studies [22,24,25,26,27,28]. Figure 1 shows the correlation plot between the proportion of RCTs in ten mask meta-analyses and their respective pooled odds ratio or risk reduction ratio. The lower the proportion of RCTs, the greater the effect of masks (lower OR or RR). In fact, there is a strong association between these two variables (Pearson correlation coefficient = 0.88). It should be noted that two systematic reviews [18,23] that included only RCTs in their analysis and the meta-analysis conducted by Wang et al 2020 that contained both RCTs and observational studies [30] found no statistically significant effects.

Additionally, real-world (ecological) observations greatly contradict the overoptimistic assessments made by meta-analyses based on observational studies [21,22,23,24,25,26,27,28], whose combined OR was 0.28. To illustrate this point consider Spain and Italy, which had, respectively, 95% and 91% of their population reporting always wearing a mask while leaving home, the highest levels of mask compliance during the COVID-19 second wave (October 2020 - March 2021) in Europe [31].As of that period, Italy and Spain ranked 20th and 18th in terms of COVID-19 cases among 35 European countries. If the actual OR of masks had been 0.23, the Spanish and Italian population should have been among the countries with the lowest COVID-19 infection rates. While western countries saw a rise in COVID-19 cases, Japan, known for high mask adherence, saw far fewer as early in the pandemic [32,33]. However, between January 1st and December 31st of 2022 Japan underwent a 15-fold increase in COVID-19 cases (from 1.73 million to 29.23 million cases), even though mask compliance was still above 85% [34]. While these uncontrolled observations do not prove that masks do not contribute to viral transmission, they do indicate that most observational studies reported unrealistically high effectiveness levels.

1.4. Randomised Trials with Positive Results of Masks and Their Statistical Significance

Two RCTs showed statistically significant results in favor of masks. The Bangladeshi study, which showed a small positive effect of surgical masks on SARS-CoV-2 transmission, did not provide exact p values, but stated that the overall effect of surgical masks (11% RR) was statistically significant at the 5% level. Among the 9 RCTs reviewed by [18], only one showed a significant level of confidence (p = 0.0238) in one of the analyzed interventions - medical masks were found to reduce clinical respiratory illness (but not ILI or lab-confirmed viral illness) [35]. Nonetheless, it is worth reminding that a p value just below 0.05 indicates that there is a 26% chance that the result is a false positive [36].

1.5. On Masks and Condoms

In both popular and scientific literature, masks are frequently referred to as physical barriers against respiratory viruses, much like condoms act as physical barriers against sexually transmitted diseases ([37,38,39];https://time.com/6232830/why-masks-still-matter-covid-19/). However, despite the widespread belief in this analogy, condoms and masks are not comparable, mainly because these two protection tools provide completely different levels of protection. In the cRCT with the most positive results [7], masks significantly reduced the prevalence of COVID-like symptoms by 11.6% (a 1.13 risk reduction). On the other hand, a Cochrane review on condom efficacy concluded that condoms are effective to a degree of 80%, i.e., a 5-fold risk reduction [40]. Condom effectiveness was estimated from observational studies, in which information about individuals inadvertently exposed to HIV while using or not using condoms was collected. As a consequence, obtaining clear evidence of effectiveness, that can be provided by randomized trials, is not possible due to ethical considerations (especially in cases of incurable diseases). As an appropriate proxy for STD (sexually-transmitted diseases) transmission, randomized trials have been conducted on condoms as a means of preventing pregnancy. According to the pooled estimate of a Cochrane review [41] on the effectiveness of non-latex versus latex male condoms for contraception, latex condoms were 97.8% effective against pregnancy, a failure rate of 2.2%, in 11 separate studies. Since the non-latex condom test group was always compared to a latex condom control group, these randomized trials did not pose any ethical concerns. Participants in these trials reported incidents of condom breakage and slippage, which are considered bona fide indicators of condom failure [42]. In contrast to mask trials against respiratory viruses, which use definitive endpoints, such as seroprevalence or viral nucleic acid detection, this alternative condom efficacy endpoint is less stringent, since not all broken or slipped condoms cause pregnancy. Therefore, the overall effectiveness of condoms in preventing pregnancy is likely to be higher than 98%, further exacerbating the difference between masks and condoms. In light of the overwhelming difference between the effectiveness of masks and condoms, which, in the case of masks, results in very low risk reduction (11.6% or 1.13-fold according to the most favorable cRCT [7], the argument that masks are comparable to condoms is disproportionate given that condoms result in a risk reduction > 50-fold (>98%).

1.6. Masking as an “Alternative Therapy”

As a general rule, the better the quality of the study (e.g., observational vs. randomised trials), the lower the reported efficacy of masks. This rule also applies to clinical trials of "alternative therapies." One example is homeopathy, a widely known and controversial alternative therapy that is used by thousands of doctors around the world and that had a market size valued at US$ 6.2 billion in 2020 (https://www.precedenceresearch.com/homeopathic-products-market). Homeopathy is based on two principles: (1) “simila similibus curentur” – “like cures like”, which holds that substances that cause similar symptoms in healthy individuals can cure diseases, and (2) the law of infinitesimals, which states that a drug’s curative power can be increased by succussion, a sequential process of ultra dilutions. Consequently, the second premise of the homeopathic canon violates fundamental principles of chemistry and pharmacology, rendering it highly unlikely to be valid. The field of homeopathy has, however, seen hundreds of clinical trials over the years. Shang et al. examined 110 homeopathy randomized or quasi-randomized control trials and 110 matched conventional medicine trials [43]. They found out an average odds ratio of 0.88 [0.65–1.19] in the eight higher quality homeopathy trials, while in their conventional counterparts the odds ratio was 0.58 [0.39–0.85]. When looking at the lower quality trials the odds ratio of both homeopathic and conventional treatments improved (became smaller), but more predictably so in the former. For instance, when not double-blind were confronted with double-blind studies, the reduction in OR were higher and significant for homeopathy (0.44 [0.22–0.87]) but not so for conventional drugs (0.63 [0.36–1.11]). It is evident thus that there is an inverse correlation between study quality and positive effects in placebo-controlled trials of homeopathy. Clinical trials showing positive results for homeopathy also had a small quantitative impact and a wide confidence interval. Why did some RCTs show a positive effect for homeopathic interventions? Those might be the result of a combination of methodological deficiencies and biased reporting. A similar pattern emerges when mask trials are examined, with observational studies showing better outcomes for masks than higher quality RCTs and cluster-RCTs, as well as when comparing laboratory-confirmed viral infection with the less well-defined clinical illnesses. Even when supported by randomized trials with statistical significance, small effects in conventional or alternative therapies are dangerously susceptible to biases. They should not be taken as evidence of causality, and certainly should not shape public health policy. In contrast, effective interventions such as condoms for the prevention of pregnancy and STDs, and vaccines and antibiotics for the prevention and treatment of infectious diseases, usually provide robust conclusive results. Take, for example, two Cochrane meta-analyses that examined the use of antibiotics. One of them analysed the effectiveness of oral or parenteral treatments with co-trimoxazole, penicillin and ampicillin for severe pneumonia in children, where the success rates were of 80-90% [44]. The other meta-analysis reviewed the use of doxycycline, tetracycline or rifampicin against scrub typhus with success rates of 95-100% [45]. Many other randomized controlled trials of antibiotics have been hitherto performed, with similar success rates. In contrast, homeopathic treatments have no success rates even remotely similar to antibiotics. Even though not all antibiotics are effective against all bacterial infections, there is no doubt that they have revolutionized medicine by providing fast and effective chemotherapy against deadly bacteria.

1.7. Masking Priors

It is true that, from a Bayesian perspective, masking should be viewed differently from homeopathy. Since homeopathy was built on an unlikely theoretical framework, particularly the infinitesimal rule of ultra-dilutions, homeopathic drugs have a very low prior probability. Conversely, intuition, common sense, and most importantly, laboratory experiments provide masks with a relatively high prior probability. It is undeniable that masks are capable of filtering particulates, even if the actual filtration effectiveness of different mask types varies widely across studies (in large part due to variation in experimental design and particle sizes analysed) (https://www.cdc.gov/coronavirus/2019-ncov/science/science-briefs/masking-science-sars-cov2.html). Surgical or cloth masks, which are used by the vast majority of people, provide only 10-12% filtration efficiency. Respirators (P2/N95) are more efficient, but none of them achieve more than 60% filtration, even under optimized laboratory conditions [46]. In addition, a small fractional leak area of 2% can severely reduce mask filtration efficiency by 75%, especially for particles smaller than 5 mm in diameter [47]. Even before 2020, several credible randomized controlled trials demonstrated that masks do not perform as well in practice as in the laboratory [18]. By utilizing meta-analyses of randomized trials as a prior instead of mechanistic evidence, the prior probability of masks comes close to zero, not so different from other alternative therapies. In other words, masks are plausible interventions based on laboratory experiments, but their continued failure in clinical trials to reduce viral infections in healthcare settings and in the community undermines their usefulness, making them similarly ineffective as alternative therapies. In this regard, the main differences between masking and classical alternative therapies is that the former has been widely supported by the mainstream medical and academic establishment and that alternative therapies have never been mandated by law.

1.8. Mask in the Lab – Filtering Efficacy as a Secondary Outcome

The ability of masks and respirators to filter viral particles in laboratory conditions has been examined in a great number of in vitro mechanistic studies. Many of these studies were published during the COVID pandemic, after masks were mandated worldwide (see [46,47,48,49,50] for a few examples). In addition, pictures and remarks taken from or based on such studies have been widely used to promote masking in the public. The enthusiasm on the apparent efficacy of mask filtration resulted in calls for disregarding RCTs (Muller 2021) in favor of what some call EBM+ (evidence based medicine+) where the ‘+’ refers to the addition of non-clinical evidence such as in vitro experimentation (mechanistic evidence) [51]. Their argument goes as follows: “the complex nature of the pandemic, characterized by multiple variables interacting dynamically with a high degree of uncertainty, urgency and threat to public health, requires the use of mechanistic evidence once randomized trials are difficult or impossible to conduct”. There is no evidence to support the assertion that randomized trials are difficult or impossible to carry out. Mask RCTs can be conducted without any hindrance from an operational perspective, as they have been conducted in the past, even during epidemics. On the allegation that conducting mask RCTs during a pandemic is unethical, we should remember that all previous randomized trials have consistently shown no effect of masks in reducing viral transmission, thus performing new RCTs would not involve any ethical concerns, as the control group would most certainly not be at risk. Additionally, the preoccupation with the severity and uncertainty of the pandemic should reinforce, rather than undermine, the need for good and reliable data. This is not the time to disregard the best tool available to the biomedical sciences, the RCT, to distinguish between working therapies and bogus or ineffective ones. The conflict between low quality data, supported by observation and common sense, and high quality data permeates the medical literature as shown by Fanaroff et al. who describe 31 heart-related conditions in which common sense or observational evidence predicted positive outcomes, only to be contradicted by robust clinical trials [2].

Whether a specific therapy or intervention should be implemented cannot be determined solely by mechanistic evidence. The best that laboratory results can do is provide the theoretical foundation for a successful clinical trial. Furthermore, the biomedical sciences is currently experiencing a reproducibility crisis, especially in pre-clinical research, where high-impact journals publish studies that yield false positives that could not be replicated [52,53]. Therefore, it is unwise to use pre-clinical research that is prone to false-positive results to justify mask use. The fact that a particular drug or therapy is successful in the laboratory does not necessarily indicate that it will be successful in the clinical trial. Many promising drugs with state-of-the-art mechanistic explanations have been withdrawn after failing randomized trials. In reality, less than 14% of pre-clinical potential drugs make it to the market [54]. Different therapeutic groups have varying success rates. Oncology-related drugs, that account for the majority of clinical trials, have a success rate, i.e. the percentage of drugs that are eventually approved by the regulatory authority, of 3.4%. In contrast, biologics such as vaccines are approved at a rate of 33.4% [54]. Some particular therapeutic fields are plagued by an overabundance of failures. Until 2008, there were over 200 failed Alzheimer’s drugs and candidates [55] and their number keeps increasing. The same is true for other therapeutic areas such as obesity [56] and most other medical fields [54]. All of these potential drugs passed pre-clinical trials with flying colors, but failed at one or other stage of the clinical trials. Similarly, mask testing under laboratory conditions may have shown filtration efficacy (although the degree of efficacy is unclear), but clinical trials (RCTs and cRCTs) have repeatedly demonstrated that they are not effective in reducing viral transmission in the community and among healthcare workers. Real-world situations are often considerably more complex than mechanistic experiments conducted under controlled conditions in laboratories, especially since we do not fully understand how respiratory virus transmission works.

Throughout the COVID-19 pandemic masks have been promoted as an important tool to reduce or even halt the spread of SARS-CoV-2 in the population. Many governments mandated the use of masks in public by law. However, even before the pandemic, the best evidence available - randomized trials, were already showing that masks are likely ineffective in curbing respiratory viral transmission. Additional high-quality data produced throughout the pandemic reinforced that conclusion. In spite of that, with a few exceptions [57], most critical reviews or opinion articles have never questioned the effectiveness of masks in the COVID-19 pandemic. While intuition and laboratory experiments suggest that masks are plausible interventions, their continued failure in randomized clinical trials undermines their effectiveness.