Which experts should we trust?

Examine, a weekly newsletter written by national science reporter Liam Mannix, is sent every Tuesday. Below is an excerpt – sign up to get the whole newsletter in your inbox.

Earlier this month, Joe Rogan – podcaster, UFC commentator, marijuana enthusiast – interviewed Dr Robert Malone about COVID-19 vaccines.

Dr Malone told Rogan’s enormous audience that data showed an “explosion of vaccine-associated deaths”.

Joe Rogan has come under scrutiny for views aired by guests on his show.Credit: Getty

He also claimed hospitals are financially incentivised to cite COVID-19 as cause of death; said a state in India had beaten COVID-19 with Ivermectin (but was covering it up); and said people who were vaccinated are more likely to get Omicron than the unvaccinated.

Quite a lot to take in! I’ll leave the fact-checking to others – the podcast Science Vs, which has led the campaign against Spotify’s enabling of misinformation, does a great job here – but Dr Malone’s thoughts raise a tricky issue.

You see, Dr Malone does have some claim to expertise. He did do important early work on developing mRNA vaccines back in 1989 and 1990, including really fundamental stuff about delivering mRNA into cells using lipids.

This raises an important question for people reading about science – and, really, about anything: Which experts should we trust? Who do we believe, and who do we ignore? And … what do we do when experts disagree?

The big pyramid of who knows best

Last week, we talked about the pyramid of scientific evidence.

This pyramid explains how science thinks about levels of evidence. Randomised controlled trials are the highest-quality form of experiment scientists can do; reviews that pull together multiple randomised controlled trials are even more powerful.

The evidence pyramid.Credit: The Logic Of Science

All the way down the bottom of the pyramid is expert opinion. Why?

Good science counts on reducing the effect of humans, who are prone to errors and biases. An expert opinion is – hopefully – grounded in evidence, but it’s also influenced by the expert’s own bias and judgment.

So, we should place evidence above expert opinion. But that does not mean there is no place for experts, says Professor Joan Leach, director of the Australian National Centre for Public Awareness of Science. Experts take the data and make a judgment about what it means.

“Expertise isn’t just about doing what the data tells you, it’s also about judgment. And that’s the tricky bit,” she says.

“You can have all the best evidence, controlled clinical trials, the highest peak of the evidence – but it still may not tell you what you need to know to make a policy decision.”

Who gets to be an expert?

In science, working out who holds expertise is down to published papers. They are “the coin of the realm,” says Professor Leach. Who is publishing the best work in the best journals?

Scientists know who the best of the best are, and who the bozos are, says Dr Darrin Durant, an expert on expertise at the University of Melbourne. It’s a bit like how a plumber knows who the best plumbers are, or how you know who are the real experts in your field.

Beyond published papers, there is status. Does the expert hold a recognised title at a leading university?

“Are they new, or experienced? Have they been fired? Have they maintained employment in their field? Do others cite them?” says Dr Durant.

When experts disagree

Usually, expert disagreement happens far from public scrutiny, often via polite letters in academic journals. COVID-19 has exposed that debate to the public.

Burnet Institute epidemiologist Mike Toole – a very nice fella on the phone – labelled a tweet from Nick Coatsworth “disgraceful”, and accused Professor Fiona Russell of being “disingenuous” (my colleague Deborah Snow covered the stoush here). Kirby Institute virologist Greg Dore had to weigh in to break the whole thing up.

Personally, I think this sort of stuff is unedifying. Before the pandemic, Twitter used to be a great place to watch scientific debate unfold. Now even science-Twitter is often a deeply ugly space, full of vitriol and personal attacks. I don’t think this is good, and I don’t think it makes us any smarter; that’s why I’ve stopped tweeting.

Professor Leach disagrees. “It think it’s great. We should embrace this – it shows the best part of science. And frankly they do eventually land on a consensus. But it’s through that argument,” she says.

“Does it leave the public confused? Maybe. But that’s not all down to the scientists, that’s down to science communicators, to journalists, to not cover these things like it’s a boxing match but as a reasoned debate.”

Making sense of it all

Clearly, this is a problem without easy solutions. The experts I spoke to were broadly pessimistic about the ability of the public to sort expert from expert.

Nevertheless, we have to try. Indeed, much of my job as a science journalist covering the pandemic is working out who to give a platform to.

In that spirit, let me leave you with a general guide I use to work out whose views to cover. I welcome your critique!

How much weight should I give this expert? Questions to ask:

Does this expert hold a position of expertise directly relevant to the issue?
For example, don’t ask epidemiologists about vaccine mechanisms.

Has this person published research that is directly relevant to the issue?
When writing about COVID, for example, I prefer experts who have published on COVID-19 over those who have not.

Does this person have rank or status within their field?
For example, are they a member of an expert group? Are they on the executive of that group?

Take the example of Professor Fiona Russell. You’ll see me quote her regularly in my stories on vaccines and children’s health.

Why? First, Russell is a recognised domain expert. Second, she publishes weekly reports on vaccine efficacy for the World Health Organisation. Third, she is deputy chair of the Vaccination Special Interest Group of the Australasian Society for Infectious Disease. Oh, and she’s regularly published in authoritative journal The Lancet on children and COVID-19.

Is this person clearly speaking to the evidence?
I prefer experts who can point to data that supports their claims.

Is this person respected by peers within their field?
This one is much more difficult, as it needs a lot of background interviews, but I think is very telling and I give it a lot of weight.

As one example, I am yet to find subject-matter experts in mathematical disease modelling who do not hold the Burnet Institute in very high esteem. Perhaps they are unwilling to speak out? Nevertheless, that makes it hard for me to buy the critique of their modelling from non-experts.

Does this person have an obvious agenda that would bias their judgment? Do they hold other biases?
This is more about tempering and contextualising the advice of experts. Professor Devi Sridhar notes, for example, there are three camps of COVID-19 experts: those who remain terrified of the virus; those who want us to let-it-rip; and a middle-ground group. Working out where an expert fits on this axis is important to understanding their judgement of the evidence.

So … should we trust Dr Malone? What about Joe Rogan?

We can answer this question more easily now. First, other than on matters of UFC or marijuana, Mr Rogan is not a recognised expert, so we don’t need to worry too much about his opinions.

Dr Malone is a different kettle of fish. He does have some level of expertise in basic mRNA mechanisms, although his public work was a long, long time ago. More recently, he has worked as a vaccine consultant. For the layperson, he’s not easy to dismiss.

Here we can look to the opinion of other experts in the field – remember, they have the best idea of who is and is not an expert. The 270 scientists who signed an open letter claiming Dr Malone’s interview was full of “false and societally harmful assertions” gives us a good sense of how other experts feel.

“If your critics are your fellow specialists, that’s a big problem,” says Dr Durant.

Then there’s what Professor Leach calls secondary sources: high-quality journalism that fact checks Dr Malone’s claims, and finds them outlandish and wrong.

“There was a flight to quality during the pandemic. People went back to quality broadsheet newspapers. Because these people they felt were trustworthy to make an interpretation. That’s why I think we need more science journalists.”

If you liked this article, consider signing up to receive Examine, a free newsletter, each week in your inbox.