Simine Vazire is one of the ARC’s 100 new future fellows

A University of Melbourne professor will take a critical look at the quality of research in universities and suggest how to improve it.

Professor Simine Vazire from the University of Melbourne.

Simine Vazire’s starting point is that the evaluation of research published in scholarly journals needs an overhaul and now she has just over a million dollars and four years to make the case.

“There are a lot of temptations and pressures on science and scientists that require a few more safeguards in place than peer review currently provides,” says the University of Melbourne psychology professor who was among the 100 mid-career researchers awarded a future fellowship last week by the federal government’s Australian Research Council.

These pressures include status bias, when there is potential for work from well known researchers to be viewed more favourably, and the overemphasis placed on being seen in what are regarded as high impact journals (which Professor Vazire argues is “not the same thing as doing quality work”).

Another pressure is the financial imperative of publishing interesting papers regularly.

“For some scientists, it’s their livelihood. It’s not just extra cash to fund more research, but also their salary, their ability to put food on the table and hire their staff and so on. A lot of how we’re evaluated, our ability to get jobs, to be promoted, to get awards depends on finding exciting discoveries,” she says.

While scientists are “not nefariously cooking the results” the push factor is accompanied by a pull factor: “the publishers, the journals, the editors or reviewers, they also want to see something exciting”.

Her interest in creating “a public, transparent, and valid process for producing and sharing expert evaluations of individual papers” springs from her experience as a journal editor over more than a decade.

“Seeing how peer review works and the kind of imperfections in it and how much responsibility is placed on peer review for being a quality filter … has made me think that we could do a lot better.”

She does not claim to be on her own in pushing for reform for scholarly publishing: thousands of scientists signed the Declaration on Research Assessment, now nine years old; and in Australia there are moves afoot in the field of legal scholarship led by academics such as University of Sydney’s Jason Chin; and there is the campaigning work of Bond University epidemiologist Paul Glasziou in improving quality in medical papers.

“I want to supplement careful thinking and reasoning about what we should do in peer review with empirical research on what works,” Professor Vazire says.

To that end, her team will recruit relevant experts to peer review papers, garnering at least three reviews for 1200 psychology papers that have already been published and examine what their comments and ratings reveal.

“The experts would be asked to rate the paper on multiple aspects of quality because scientific papers can be really good in one way but fall short in a different domain. And I want to be able to capture that nuance and the richness of the different ways that scientific findings can be good or bad or trustworthy or not,” she says.

“Validating it would include things like: if you collect multiple experts readings of the quality of a paper, how well do they agree? The goal isn’t perfect agreement necessarily, but too little would be a sign of a bad measure.”

Her lab has been working on another project for the past few years, coding relatively objective features of papers such as sample size, the types of methods used and the quality of the statistical work.

The aim is to come up with a new measure, Quality Factor for Psychology, combining the experts’ validation parameters with the nuts and bolts features to produce a database that could be of use to academic and general audiences when evaluating research. If all goes well, Professor Vazire hopes it will be adaptable to fields outside her own.

She calls it “getting uptake … making sure that other groups who are organising peer review, for example, scientific journals – incorporate this measure (once it’s been validated) into the process so that it’s not just our project that’s generating data on the quality of individual papers.

“So we don’t just waste all the information we get from the reviewers, the expertise, the hard work, but we capture that information, quantify it and share in a way that is useful to as many different people as possible,” she says.

Education Minister Alan Tudge says the $93m being spent on the 100 future fellows would ensure that Australia stays in the forefront of research.