Ivermectin, COVID-19, and making sense of scientific evidence

Examine, a weekly newsletter written by national science reporter Liam Mannix, is sent every Tuesday. Below is an excerpt - sign up to get the whole newsletter in your inbox.

Few scientific issues seem to get people as excited as ivermectin, a potential treatment for COVID-19. I receive dozens of emails about it every week.

“People are dying,” they write. Ivermectin could help. “I know there is a huge concerted effort to suppress any information about its efficacy.”

Indeed, there does seem to be an overwhelming amount of seemingly scientific evidence available on the internet showing this drug works. However, until this point, nearly all the evidence on ivermectin has been of such low quality we simply cannot draw any conclusions from it.

A new study may soon change that. More on that in a sec.

First, we’re going to look at a fundamental skill of science: understanding evidence.

The shortcut method

Deciphering evidence is really hard work; scientists spend much of their time arguing over what a piece of data really means.

“I think it’s hard. If I were a layperson, I don’t know if I would try to assess the evidence in that way. The stuff I look for, it took quite a while to learn,” says University of Wollongong epidemiologist Gideon Meyerowitz-Katz, who has spent much of the past 18 months interrogating the evidence around ivermectin.

Luckily for us, there are shortcuts to quickly look up the highest-quality evidence.

Perhaps the best of these is Cochrane. It’s an international non-profit bringing together top scientists to review evidence and produce easy-to-understand reports, using a special Cochrane method that eliminates sources of bias, like the insidious role of drug company funding.

“Generally, if there’s a Cochrane review, that’s the gold standard,” says Mr Meyerowitz-Katz.

If you have a question, Cochrane often has the answer - even for some unexpected stuff. They reviewed electric toothbrushes and found that, yes, they really do work better than the manual kind (which led me to buy one)! Start your search here.

Australia’s federal government wisely funded Cochrane to set up a COVID-19 taskforce here, made up of representatives from across Australia’s medical societies. They are independent and unafraid; willing to point out a drug the government has purchased to treat COVID-19 might not work.

For these three reasons - independence, Cochrane methodology, representation across the medical spectrum - we can have a very high degree of confidence in their findings.

Compare that to c19early.com, one of the leading ivermectin websites. Put on your critical thinking hat, and ask: is an independent, government-funded, Cochrane-using panel made up of scientists drawn from across Australia’s leading institutions, or… a website put up anonymously... best placed to scrutinise the evidence?

Every week - seriously, every week - the Cochrane taskforce reviews all the new evidence generated on how to prevent and treat COVID-19. All this goes online. You can look up any treatment you want here. Just hit Ctrl-F on your keyboard.

Here’s its recommendation on ivermectin: Do not use ivermectin for the treatment of COVID-19 outside of randomised trials with appropriate ethical approval.

Click on ‘research evidence’ and you can see a collection of all the published ivermectin data, sorted by outcome. But the more important thing to look at is the columns on the right.

Certainty of evidence takes into account the quality of the study. A study might claim to find ivermectin works, but if the quality is low, we cannot have faith in that finding. The evidence for every single clinical outcome for ivermectin is either low or very low quality.

There is simply not enough evidence to show whether ivermectin is helpful or harmful for patients with COVID-19.

The longer method

Let’s say, like crossbencher Craig Kelly, you don’t want to take the National COVID-19 Clinical Evidence Taskforce at its word. Good for you! We should always be sceptical of knowledge claims, particularly those made by authorities.

Crossbench MP Craig Kelly is a strong supporter of the use of ivermectin as a COVID-19 treatment.Credit: Alex Ellinghausen

So, how can you as a layperson work out if evidence is of high quality or not?

This is not easy, warns Mr Meyerowitz-Katz. The pandemic has led to a dip in general research standards, with low-quality evidence published in what should be high-quality journals. And ivermectin, for some reason, seems to attract fraudulent research. Multiple important ivermectin studies have been withdrawn after concerns about data manipulation and plagiarism were raised.

“There’s this issue in science that everything is based on trust, everything. Without trust the whole system falls apart. If a paper is published, you assume it at least happened.”

This is the reason science needs people like Mr Meyerowitz-Katz running a fine eye over the evidence - because some problems are hard to spot.

However, as laypeople, there are a few basic things we can do to test the quality of information.

The easiest clue: look at where the evidence is published.

Scientists publish their work in academic journals. To get published, your work generally needs to pass peer review: a panel of experts will check it over to make sure it’s solid.

As a general rule of thumb, the bigger the journal the harder it is for a paper to pass peer review. Nature, probably the world’s leading scientific journal, rejects many times more papers than it publishes. At the other end of the spectrum, small journals might only do a cursory peer review (or no peer review at all).

You can look up the quality of a journal here. Look for a journal’s ‘quartile’. Anything below 2 should be treated with serious scepticism.

Let’s pick the first study on c19early.com. It’s a preprint. That means it hasn’t passed rigorous peer review yet. While that does not automatically mean it’s of low quality, we should be very sceptical.

Now, let’s think about what good scientific evidence looks like. To know if ivermectin works, you’d need two large groups (they need to be big to eliminate the role of luck) of people who are almost exactly the same. Same age, same underlying risk factors. Half get ivermectin, half get a placebo. Known as a randomised controlled trial, this is the highest form of scientific evidence. Scientists used this technique to discover the cheap steroid dexamethasone can help save the lives of people seriously ill with COVID-19 (ivermectin’s pundits are quick to claim drug companies are covering up its benefits… but never mention the big pharma’s failure to cover up dexamethasone).

Compare that to the top study on c19early.com. It claims it shows a 19.6 per cent reduction in the risk of death if a patient is taking ivermectin. Wow! But the study itself barely mentions ivermectin. And it does not tell us who the patients who got ivermectin were. How old were they? What underlying conditions did they have? Why did the doctors give them ivermectin, and not to other patients? How do they compare to the patients who died?

We can learn almost nothing from this study. Sadly, this is the problem with websites that purport to pull together heaps and heaps of studies and claim to make a conclusion - we, as laypeople, don’t have time to check each one. This is why I, as a science journalist, turn to groups like the Evidence Taskforce, who are paid to read these papers.

Let’s turn, now, to a study that is actually quite good.

It’s called the TOGETHER trial. It’s a randomised controlled trial, and it’s very large. It publishes all its trial methodology online for everyone to read. Mr Meyerowitz-Katz describes it as a “masterpiece of science”.

The full findings haven’t been published yet, so we should treat them with a grain of salt. But the conclusions are available via a PowerPoint slide. They found essentially no benefit to ivermectin.

(If you’d like to read more about ivermectin and understanding scientific evidence in more depth, I highly recommend Mr Meyerowitz-Katz’s website).

If you appreciated this article, consider signing up to receive Examine, a free newsletter, each week in your inbox.