Closing the Gap: beware hazards of metrics gone mad

Not everything that can be counted counts and not everything that counts can be counted.

The government’s ambition to close the gap between indigenous and non-indigenous Australians has been a major failure, an all-too-common triumph of hubris over reality. As The Australian reported this week, governments have spent more than $130 billion on programs that help meet the six, now seven, targets laid down by the Rudd government in 2008 to improve health, education and employment outcomes.

Yet even by its own assessment, only one of the targets — to halve the gap in the share of 20 to 24-year-olds enrolled in Year 12 or equivalent education — is “likely to be met”. And let’s be frank: that’s a tick-a-box target. Imagine if actual learning could somehow be tracked. And that’s precisely the point — it can’t. Numerical measures can obscure and distort as much as clarify.

The actual outcomes for the indigenous Australians Kevin Rudd was worried about in 2008 are probably far worse than the latest statistics. That’s because the number of Australians counted as indigenous has been growing far faster than demographics would imply.

Between the 2011 and 2016 censuses, the number of Australians identifying as Aboriginal ballooned more than 18 per cent, or by more than double the rate of population growth, to 649,000. Immigration can’t explain that, and Aboriginal birthrates are only a little above those of non-indigenous Australians. An increased “propensity to identify” as indigenous, as the Australian Bureau of Statistics calls it, explains a big chunk of the rise. And that invalidates the whole exercise if the pool of “new” Aborigines is healthier.

Indeed, the whole Closing the Gap enterprise points to a wider problem: the hazards of metrics.

Economists have always known how statistics become dubious once they have political or financial impact — Goodhart’s law. But a highly readable new book, The Tyranny of Metrics, by US history professor Jerry Muller, is a timely reminder of how such knowledge has been sidelined in the quest to monitor all manner of procedures and outcomes throughout public and private enterprise, with sometimes dubious consequences.

“Metric fixation,” Muller says, reflects the idea that complex outcomes can be boiled down to numerical form to improve transparency and sharpen incentives for efficiency. But this undercuts judgment, context and experience and, when applied to pay, “simple-mindedly assume that people are motivated only by the desire for more money”. Not everything that can be counted counts; not everything that counts can be counted.

“In situations where there are no real or feasible solutions to a problem, the gathering and publication of performance data serves as a form of virtue signalling,” he writes, singling out the worldwide push to lift standardised test scores of students from lower socio-economic backgrounds. In Australia, as in the US, a battery of routine tests reveals students from poorer backgrounds perform worse than those from upper-middle-class families, leading to calls for more funding to redress the disparity.

The extra $19bn the federal government is showering on schools across the next decade is a response to such education metrics. But closing the “achievement gap” is extremely difficult, however much money is thrown at the problem. “Pupils who are brighter, more curious and more self-controlled tend to be the offspring of people who are themselves relatively bright, curious and self-disciplined,” writes Muller.

Universities have hitched themselves even more strongly to the metric bandwagon. Beyond the damaging target to enrol ever more students — which fuels a costly “arms race” benefiting larger universities — higher education has spawned so many metrics that the cost of administrators and bureaucrats now outweighs academics.

A social sciences academic at one of the elite Group of Eight universities says hers has a “metrics obsession”. “It’s extreme, and we talk about it all the time, more than we should, and much more than our research that we’re supposed to be doing,” she tells Inquirer.

“The metrics about number of publications skews us towards less risky research and we avoid blue skies thinking,” she adds, noting the compulsion to attach percentages to any contributions to jointly authored articles.

Academics in Australia are judged in large part on the number of articles they publish, adjusted for the perceived quality of the journals they appear in, and how often they are cited by other academics.

“If the incentive system rewards speed and volume of output, the result is likely to be a decline in truly significant works,” says Muller.

Hospitals worldwide are also increasingly gripped by metrics. In New York state the collection of physician “report cards” on post-surgery mortality rates did, indeed, improve measured performance, but later studies found this was because surgeons avoided more difficult operations. “Patients whose operations are not successful may be kept alive for the requisite 30 days to improve their hospitals’ mortality data, a prolongation that is both costly and inhumane,” Muller writes.

Such data isn’t published in Australia but metric creep is evident. A physician at a major Sydney hospital points to “politically expedient targets such as elective surgical waiting lists and emergency waiting times”. About 80 per cent of patients must be discharged or admitted within four hours. “That might sound sensible but leads to all sorts of pressures to push patients out or in before truly determining what the problem may be,” he says.

Moreover, ambulances in NSW are allowed to leave patients “literally in the corridors” of emergency departments, he explains, so they can get on with the next job and meet paramedic availability targets. “It would be better for the patients if the ambulance service were not discouraged from driving a bit further to the next hospital that may have capacity on any given day — obviously not for urgent cases — but this would be a much better use of resources,” he says.

In an effort to save money, health bureaucrats in Australia now pay hospitals “efficient prices” for services performed in them, but this can entrench waste, says the Centre for Independent Studies’ Jeremy Sammut. The prices are based on the average cost of performing services at efficient and inefficient hospitals. “It’s an attempt to mimic the private sector but these prices give an incentive to hospitals to over-service to ensure they maximise their budgets,” he says.

Targets based on crime statistics are also fertile ground for metric malfunction. Changes in crime rates can reflect a lot more than actual crime, from police funding to officers’ proclivity to record particular crime and enforce certain laws.

Geoffrey Brennan, one of Australia’s most distinguished economists, says measures have their place. “The relevant comparison is whether measures are better than no measures,” he tells Inquirer.

Few industries are more output-based than journalism, yet metrics aren’t (at least to my knowledge) yet routinely used to compare journalists. Editors make a qualitative judgment, based on a range of factors, about workers’ quality. How many page 6 news articles is this short essay worth? There’s no answer.

Where does all this leave indigenous Australians? The government should acknowledge that Closing the Gap has failed and dump the seven specific metrics. Rather than obsessing over statistics, bureaucrats could focus on improving actual outcomes; reality can’t so easily be distilled into a set of consistent numbers.