Humans must have final say over decisions made by artificial intelligence, cyber expert Rachael Falk says

Humans must have final say over decisions made by artificial intelligence, cyber expert Rachael Falk says, as she calls for greater public education and oversight of AI.

Cyber Security Co-operative Research Centre chief executive Rachael Falk.

Humans must remain central to decisions made by artificial intelligence, and businesses need to be aware of the risks posed by AI, including harm to customers and cyber security threats, according to cyber expert Rachael Falk.

Ms Falk, chief executive of the Cyber Security Co-operative Research Centre, said the world was “on the precipice’’ of an AI-led revolution, yet many Australians had little understanding of how AI worked and where and when it was used.

Citing the disastrous Robodebt program, which raised wrongful debts against Centrelink clients by using computerised data-matching to average out their incomes, Ms Falk said key decisions with the potential for adverse outcomes could not be outsourced solely to AI.

She said organisations and businesses should be transparent about the extent to which they used AI, and what they used it for.

“They have got to be realistic about the threats and transparency with their customers,’’ she told The Australian. “So if they’re going to use AI, they’ve got to understand what the impact is and the potential harm, and mitigate that.

“If you’re a financial service organisation and you’re wanting to completely automate … credit cards or things like that, you’ve got to be transparent.

“If computer says no, when are you going to have a human intervene in those decisions?

“We … need to be really cognisant now, as we are standing on the precipice, of where the black box should be the sole decision- maker and where there needs to be humans in the middle.’’

Cyber expert Rachael Falk says the world is on the precipice of an AI-led revolution. Picture: AFP

In a submission to an AI discussion paper being prepared by the Department of Industry, Science and Resources, Ms Falk also raises concerns about AI’s capacity for increasing cyber security risk.

“Artificial intelligence technologies have the potential to be transformative, bringing new efficiencies to the way we work, live and do business,’’ she said in the submission.

“But with the good comes the bad and, from a cyber security perspective, while AI can serve to bolster defences, it can also be used by malicious actors to enable more sophisticated exploits and attacks.

“For this reason, the Cyber Security Co-operative Research Centre submits that in developing a plan for the safe and res­ponsible use of AI in Australia, cyber ­security must be a primary concern.’’

Ms Falk said AI technologies could produce malicious code and sophisticated fraudulent content and, in the context of deep fakes, create life-like audio, ­imagery and visual recordings that could be used for nefarious purposes.

She cited an example in Canada, where impostors using AI simulated the voices of four ­people who were in court for various matters. Four grandparents were allegedly scammed out of money, which the impostors claimed was for legal fees or bail, after they believed the voice on the phone was their grandchild’s.

Ms Falk said voice simulation was already being done, and predicted it could be used on a much larger scale.

“There’s a way you can imagine that being done, a little bit more of a sophisticated example of the common business email compromise – it will move to voice scamming and that’s a way in,’’ she said.

“That’s certainly already happening when it comes to scams.’’

In her submission, Ms Falk called for the creation of a new body to oversee the use of emerging technologies.

“When it comes to the regulation of emerging technologies, there has been a trend domestically and globally of regulating too late and trying to play catch up – cyber security is a prime example,’’ she said.

“In relation to AI technologies, which can unlock significant potential but also result in serious harms, swift regulatory action must be a key priority for the federal government.’’

She said a stand-alone body should be established to co-­ordinate AI governance across government and the private sector, similar to the National Cyber Security Co-ordinator, or the Cyber and Infrastructure Centre within Home Affairs.

A proposed emerging Technologies Regulation Centre within the Department of Industry, Science and Resources, or another suitable department, should also be considered, the submission contends.

“Human oversight must form a cornerstone of regulation, establishing checks and balances to ensure that any risks, harms or bias are detected and effectively addressed,’’ Ms Falk said.

She said AI raised bias risks, as the targeting of minority groups in the US had shown.

She also said there was scope for the government to provide guidance as to the use of AI in research undertaken in Australia.

“While the ethics of AI use in research has been a topic of much discussion globally, little attention has been paid to the security of research data and protection of intellectual property,’’ she said.

“This is an issue that also needs to be considered by university peak bodies and individual universities in relation to the protection and integrity of IP produced by academics.’’