Complex data may skew the funding

A new report suggests it may prove hard to rely on performance data as a basis for funding decisions.

The paper points out that ‘performance metrics are … marked by paradox’.

A new report outlines the high degree of complexity in the performance data of Australian ­universities, underlining the difficulties Education Minister Dan Tehan will encounter in the government’s plan to make a portion of higher education funding dependent on performance.

The study, from the Centre for Higher Education Equity and ­Diversity Research at La Trobe University, warns that a straightforward performance funding model that “simply rewards inputs and outputs rather than the value added by institutions” is likely to be unfair to disadvantaged groups of students.

“Detailed examination of existing Australian evidence reveals that many factors of performance appear uncorrelated, or indeed negatively correlated, with each other,” says the report, titled Principles for Equity in Higher Education Performance Funding.

“Such complexity highlights the need for more sophisticated and accurate metrics to identify teaching performance, learning gain, student satisfaction and employment outcomes.”

Lead author Andrew Harvey, who is director of the centre at La Trobe University, writes in a commentary in The Australian today that there are “risks and contradictions” in a performance-based funding system.

If the Morrison government is re-elected, it plans to introduce a performance element into university funding from 2020 to reward universities that do better in achieving measurable outcomes such as reducing the student attrition rate, increasing degree completion rates, and scoring higher student and graduate satisfaction rates.

Over the past few years the volume and quality of such data have improved, including the publication of student and graduate satisfaction measures, as well as salary and employment outcomes, on the federal government’s Quality Indicators in Learning and Teaching website (www.qilt.edu.au).

Performance against measures such as these could govern the distribution of up to $500 million a year of Commonwealth Grant Scheme funding for universities.

However, the paper by Dr Harvey and his colleagues shows that such performance measures are affected by complex factors and are related in ways that are apparently contradictory.

“Performance metrics are also marked by paradox,” the paper points out.

“Part-time and distance education students, for example, are less likely than other students to complete their degrees, but have relatively strong graduate outcomes if they do.

“How a performance model might account for these different, often contradictory trends presents a significant assessment challenge.”

In other examples of complex outcomes, the paper notes that ­indigenous students have relatively high attrition rates but better than average short-term employment outcomes, while the reverse is true of students from non-English-speaking backgrounds, with above-average university retention rates but below-average employment outcomes.

The paper urges policymakers to make a distinction between the raw data of “outcomes” and a more sophisticated measure of “performance” that should take into account other factors that affect students’ or graduates’ achievement level.

“A primary challenge is … to distinguish outcomes from performance, and in so doing to avoid worsening sectoral and institutional inequity,” the paper says.

It also urges that performance funding models for universities should be “student-centred”, meaning that their views are explicitly taken into account in designing performance metrics, and that information is widely available and useful to students.