Last Thursday (15) the scientific journal Trials, belonging to the Springer Nature group, published an article that reanalyzed the most rigorous study ever done regarding the effectiveness of masks to stop or reduce the transmission of the Covid virus-19. The reanalysis concluded that it cannot be said that the use of masks had a significant effect, contrary to what was concluded in the original study. The reason for this is that the original study made procedural decisions that created biases in the results comparing the mask-wearing and non-mask-wearing group.
Re-analyzed study, done in Bangladesh and published in the journal Science, had indicated in December 2021 that surgical and fabric masks had some effectiveness in stopping the transmission of the virus of Covid-15. The effectiveness would be a reduction of about 10% in the number of symptomatic and infected among those who used masks. Gazeta do Povo covered the study at the time. The study’s first author was Jason Abaluck, a professor of economics at Yale University.
The intent of the Abaluck study was to randomly choose who would wear a mask and who would not, so the results did not reflect, for example, a more zealous temperament of those who wanted to wear a mask compared to those who did not. A zealous person takes other types of precautions, so the difference in infection rates would be the zealous behavior, not the mask. To avoid this type of confounding factor, the participants were randomly assigned. This type of study is called a “randomized controlled trial” (RCT). “Control” is the group that does not receive the intervention: in this case, those without a mask.
Methodological difficulties
Abaluck and his co-authors had 340 1,000 Bangladeshi natives at their disposal to randomly allocate between the masked and the unmasked group. If this were the case, it would be an excellent sample and the conclusions would be robust. But the first difficulty is that, instead, villages were 600 randomly distributed to have a campaign for wearing a mask or not. In fact, 95 pairs, where one was placed in the control and the other in the treatment group. This has the effect of reducing the effective sample size, which refers to how many entities are actually randomly distributed. To make matters worse, village pairs were excluded because of lack of government cooperation or insufficient observation, so the remaining number is 95 pairs.
Ideally, researchers do not know who received treatment or not, and when what being evaluated is a drug, for example, in addition to the control group there is also a group that receives an inert pill (placebo). Whoever receives the pills would also not know if they are placebos or not. Thus, researchers are “blinded” and so are participants, which is why this protocol is called “double blind”. In the case of the mask study, there was no placebo group. The study employed staff to handle the work in Bangladesh. The employees were “blinded” in terms of mapping the villages and houses that would receive masks, in a first step, but in a second step they were not “blinded” as they needed to ask for consent from the people of each household.
The reanalysis notes that this decision created an imbalance between the two study groups. In the first step, staff mapped a wider range of villages to the treatment group. A difference in behavior was therefore observed in the direction of the group that received masks, despite the claim that employees were choosing blindly. In the second step, which the study admits was not blind, consent was obtained in a greater number of households, and these were households with more people, increasing the imbalance in the results. For example, households in villages allocated to the no-mask group were recorded as “no one at home” 1.4 times more often than those in the masked group, and 2.2 times more often with study participants absent when staff visited households.
The authors of the reanalysis conclude that “the difference in consent obtained by non-blinded staff is among the most significant differences between the differences in outcomes between treatment and control”. In other words, the intentional favoring or not of the employees in the observation of people with masks was one of the biggest influences on the results among those who wore a mask or not.
Dependence of voluntary reports
To make matters worse, there is one more source of bias in the study: positive Covid cases were based on participants’ self-report of their symptoms. Once this voluntary report took place, the blood test took place for confirmation. Therefore, the study cannot say anything about asymptomatic Covid, and it ran the risk of its result being influenced by mask users reporting their symptoms less than non-users.
As the reanalysis says, the raw number of the observed difference was only 15 cases: there were 1106 symptomatic individuals with test-confirmed infection in the unmasked group and 340 in the masked group. Given the large number of subjects, if few felt safe enough to wear a mask to the point of presuming that a mild cough could not be Covid, thus deciding not to report symptoms to study staff, the results would already be invalid.
Given all this, the scientists who reanalyzed the Bangladesh study concluded that the observed rates of symptomatic and symptomatic people with test confirmation “are most plausibly explained by chance fluctuations.” Furthermore, in a list of problems that typically affect the reliability of RCTs to establish cause-and-effect relationships, the study failed in all of them.
It is not all failure: the reanalysis recognizes that the campaign educational program to convince the villages to adhere to the masks was effective. More effective than attempts at imposition, such as Gazeta do Povo recounted in its original coverage. The authors of the reanalysis are Maria Chikina of the Department of Computational and Systems Biology at the University of Pittsburgh, Wesley Pegden of the Department of Mathematics at Carnegie Mellon University, and Benjamin Recht of the Department of Electrical Engineering and Computer Science at the University of California. at Berkeley, all institutions in the United States.
Just as in some studies of early treatment drugs, a lack of sufficient evidence of efficacy is not sufficient reason to claim “proof” of ineffectiveness. The main result of the reanalysis is that the hypothesis that masks did not reduce the number of symptomatic patients is more likely than the hypothesis that they did. Another comparison that can be made with early treatment drugs is that mechanisms are important: if a drug has antiviral action shown in the laboratory, this increases its chance of working. If a mask has virus-like particle filtering capabilities, as is the case with PFF2 and N95 masks, this increases its chance of working, so surgeons and other healthcare professionals are not wrong to continue wearing masks. What did not make sense was to impose masks on the population.
