Burnet Institute mask study hit with ‘expression of concern’ notice by journal Plos One

A controversial study into Victoria’s mask mandates has been dinged by the scientific journal that published it after widespread backlash.

A major scientific journal has officially acknowledged “serious concerns” about the findings of a study into Victoria’s mask mandates, more than 18 months after the paper was torn to shreds as “crap” by experts.

The journal Plos One on Friday slapped an “expression of concern” on the Burnet Institute’s controversial July 2021 paper, which purported to show that mandating masks was the “single most important control measure” that turned around Victoria’s second Covid wave in 2020 “almost overnight”.

In October 2021, experts told news.com.au that the supposed “world-first” study was riddled with basic errors and should never have been published in a major journal, blasting the paper as “crap”, “extremely lightweight” and “very, very low quality”.

Everyone is entitled to their views, but to publish a headline “it’s crap” that is attributed to the unnamed scientist, is very disappointing and unprofessional.

— Burnet Institute (@BurnetInstitute) October 21, 2021

The Burnet Institute – an influential public health body that advised the Victorian government on its Covid response – stood by its findings and described the scientists’ comments to news.com.au as “disappointing and unprofessional”.

But Plos One has now acknowledged flaws in the study with a lengthy notice appended to the paper. An expression of concern – one step above a correction but short of a retraction – is used to “alert readers of serious concerns about published work”.

“After publication of this article, readers raised a number of concerns, including about the methodology, the limitations of the study design, and whether the conclusions are fully supported,” Plos One editors wrote in the notice.

Mask mandates started in July 2020. Picture: Luis Enrique Ascui/NCA NewsWire

“The Plos One editors consulted with two members of the editorial board and a statistical advisor who advised that the study design is associated with a number of weaknesses that are discussed in the article, and which are unavoidable because of ethical issues that would be associated with a randomised controlled trial in the context of a pandemic, but that there were also additional weaknesses.”

For the study, the Burnet Institute looked at photos from The Age newspaper to conclude that mask usage rose from 43 per cent to 97 per cent after the July mandate came into effect.

Medical researcher Dr Kyle Sheldrick told news.com.au in 2021 that it was “hard to think of a worse methodology to answer this question than just looking at which photos are collected by a metropolitan newspaper”.

“I was staggered to see this was published by a major journal,” he said. “If a student came to me and said, I’m going to compare these two sets of photos and draw some conclusion about whether a policy worked, you would send them away to think about it.”

Another researcher, an eminent Australian clinician and scientist who spoke on the condition of anonymity, was equally scathing. “I agree, it’s crap,” he said. “It’s extremely lightweight. I think it’s a totally feeble article. It doesn’t have a rigorous methodology and it is weak in its scientific inference.”

Concerns were also raised about other aspects of the paper, including that it singled out the mask mandate as the key reason for the fall in Covid transmission out of all of Victoria’s sweeping lockdown measures.

Plos One’s editors acknowledged the three main issues raised with the paper – the photographic observation of mask usage, self-reported mask wearing survey data, and unmeasured confounding variables such as curfews, movement restrictions and closures.

Thousands of Victorians were fined. Picture: Luis Enrique Ascui/NCA NewsWire

On the photos, Plos One’s editors noted it was “unclear whether such images are representative of mask usage in the population”.

They also noted the Survey of Covid-19 Responses to Understand Behaviour (SCRUB) sample size was “small and may not be representative of the population”.

In response, the Burnet Institute’s authors indicated that “the sample sizes were not inappropriately small for the purpose, noting that the high-powered interrupted time series study found a highly statistically significant change in the growth rate of the epidemic that coincided with the mask mandate”.

Plos One’s editors also noted that “the study design could not exclude the possibility of contributions from unmeasured confounding variables, including the implementation of a curfew and movement restrictions on 2nd August 2020 and closure of childcare facilities, schools, and non-essential businesses on 5th August 2020”.

“According to the cumulative expert input received by the Plos One editors, the results of the published study contribute to the field of mask evaluation research, provided results are not overinterpreted and limitations are acknowledged,” they wrote.

“The Plos One editors felt that the conclusions, including those that imply causation, a direct correlation between Covid-19 cases and mask mandates, and the ability of masks for controlling epidemics, were not suitably tempered in light of the limitations of the study design. The Plos One editors issue this expression of concern to inform readers about the above considerations regarding study design and interpretation of the results.”

The Burnet Institute says it stands by the paper. Picture: David Crosling/NCA NewsWire

In a statement on Thursday, a spokesman for the Burnet Institute said the authors “stand by the results in the paper”, adding it was “unusual to encounter this sort of post-publication response and scrutiny”.

“These findings, and those obtained independently by James Trauer and colleagues came to similar conclusions about the effects a mask mandate had on reducing Covid-19 caseloads in Melbourne during the second wave,” he said.

“The main point of our paper is that the timing of the introduction of a mask mandate correlated with a statistically significant reduction in cases. We extend this to provide evidence that mask usage increased following the mandate making mask wearing the likely driver of case reduction. However, we agree that correlation does not mean causation and we state in the manuscript that ‘care should be taken in ascribing causation’ and discuss other potential indirect effects that might explain the effect of the mandate.”

The spokesman stressed that “while neither paper had any influence over the Covid-19 control measures used in the ‘second wave’ (they were after-the-fact publications), it is our view that these papers make a contribution to the now extensive body of scientific literature involving mask use in the community because the Melbourne circumstance offered a rare opportunity to examine the effects of a mask mandate introduced in the absence of other policy changes”.

“We are also aware that masks are an issue that engenders a strong response in the wider community in a way that is not seen in most other topics,” he said.

“While it is unusual to encounter this sort of post-publication response and scrutiny on methodology, we remain proud of this, and subsequent related peer-reviewed work (Saul et al) published in this space.”

Masking remains a controversial issue. Picture: Luis Enrique Ascui/NCA NewsWire

It comes after a major Cochrane Review – considered the gold standard of evidence-based medicine – earlier this year assessed 78 high-quality scientific studies, and found “wearing masks in the community probably makes little or no difference” when comparing masking with non-masking to prevent Covid.

The bombshell findings proved controversial and sparked a war of words between Karla Soares-Weiser, editor-in-chief of the Cochrane Library, and the study’s lead author Tom Jefferson.

“Many commentators have claimed that a recently-updated Cochrane Review shows that ‘masks don’t work’, which is an inaccurate and misleading interpretation,” Soares-Weiser said in a statement in March.

“[The study’s] wording was open to misinterpretation, for which we apologise. While scientific evidence is never immune to misinterpretation, we take responsibility for not making the wording clearer from the outset.”

Jefferson subsequently hit back at suggestions the apology from Soares-Weiser meant the study had been retracted.

“It was upsetting,” he told journalist Maryanne Demasi. “Cochrane has thrown its own researchers under the bus again. The apology issued by Cochrane is from Soares-Weiser, not from the authors of the review.”

Jefferson claimed Soares-Weiser, in response to media pressure, had “gone outside the normal channels and made decisions without any consultation with the authors of the review”.

“It is unacceptable,” he said. “We are the copyright holders of the review, so we decide what goes in or out of the review. We do not change our reviews on the basis of what the media wants.”

[e-mail]