Experts alarmed over chatbot harm to teens
AI companion chatbots are ‘much more dangerous’ to Australian teens than social media, experts warn, with the technology being linked to the suicide and radicalisation of young people overseas.
AI chatbots are more dangerous to Australian teenagers than social media and YouTube, with the technology being linked to the suicide and radicalisation of teens overseas, experts warn.
Australia late last year passed groundbreaking legislation aimed at preventing anyone under the age of 16 from registering to join social media platforms, including Facebook, Instagram, X and Snapchat – and this week there were calls for the popular video-sharing app YouTube to be added to the ban.
There is also a push among some experts to extend the legislation to include generative AI companions and bots, with concerns about the manipulative nature of the technology.
A Florida mother in 2024 sued chatbot creator Character.ai, claiming it contributed to her 14-year-old son’s suicide.
Megan Garcia alleged the AI chatbot app had “abused and preyed on my son, manipulating him into taking his own life”.
In 2023, a UK court sentenced a 21-year-old man to nine years in jail for breaking into Windsor Castle with a crossbow. The trial heard how Jaswant Singh Chail had exchanged more than 5000 messages with an online companion he’d named Sarai, which included discussions about how he wanted “to assassinate the queen of the royal family”.
In an interview with The Australian, leading men’s health expert and clinical psychologist Zac Seidler warned the technology was being “weaponised” against young men.
The fact this technology “is being weaponised against young men, and the fact that their likelihood and impulse control and risk taking makes them more at risk for suicide and self-harm is really, really concerning”, he said.
Dr Seidler said AI chatbot platforms, which were “popping up everywhere”, were “purposefully going to create harm” and “exacerbate symptoms of disconnection and depression over time among young men”.
“If it continues with the trends that we’re witnessing at the moment, I am really concerned that this could get very, very bad in ways we can’t even see right now,” he said.
Mhairi Aitken, a senior ethics fellow at the Alan Turing Institute, said AI technologies had not been designed in ways that were safe for children, adding the risks were, in many ways, “much more dangerous” than social media.
“The risk here is that it’s exacerbating or amplifying a lot of the risks and harms of social media,” she said.
“We should have learned the lessons … there’s a real risk that this is repeating (those same mistakes) and potentially in a worse way.”
Toby Walsh, a leading expert in artificial intelligence at the University of NSW, agrees.
He says “we should be very concerned” at the development of companion AI chatbots, adding that “social media should’ve been a practice run”.
“We’re about to run another experiment on our young children and potentially the consequences could be quite detrimental,” he said.
Professor Walsh urged politicians and regulators to closely consider banning the technology, which would “incentivise the tech companies to build age-appropriate spaces for young people”.
“There’s a lot of money at stake and there’s a lot of commercial pressure for them to do this,” he said.
“The only way to alleviate that pressure is to outlaw it if they’re not going to do the right thing.”
Meetali Jain, director and founder of the non-profit Tech Justice Law Project, said she was not in favour of banning the technology, but instead would like “to see a change in how Silicon Valley does business”.
“As the mother of two children myself, it’s really frightening,” she told The Australian.
“The more I’ve started to learn about companion chatbots, it terrifies me in ways that I never thought possible with social media.”
Ms Jain was contacted in May last year by Ms Garcia, and said the Florida mother had a hard time being believed and finding support for legal representation, and she had decided “to jump into it” and represent her, helping to draft a complaint.
Since launching Ms Garcia’s case in October in the federal courts in Florida, other families had come forward, Ms Jain said, adding that she was now representing three families who were alleging that AI companion chatbots had caused their children “pretty grievous harm”.
Asked what concerned her most about AI chatbots, Ms Jain said it was the fact the technology was being developed “under our nose in plain view, and yet we didn’t even know about it”.
She added: “I had a parent asking me a few months ago, ‘Do I know any sex addiction therapists for her 11-year-old?’”
Australian Psychological Society president Sara Quinn said she was “gravely concerned” about the potential harms of the technology for young people.
Dr Quinn called for a “comprehensive framework in Australia” and “more rigorous oversight from tech companies” to ensure that these tools were safe for children.
Speaking at the National Press Club on Tuesday, eSafety Commissioner Julie Inman Grant flagged that Australia’s social media ban would not address emerging AI harms for young people, but said the unregulated use of AI companions was of particular concern.
“The rise of powerful, cheap and accessible AI models without built-in guardrails or age restrictions are a further hazard faced by our children today,” she said.
In a statement, Ms Inman Grant told The Australian that eSafety had “been monitoring the rise in popularity” of AI chatbots and had “taken a number of steps to inform parents about the risks”.
“It is important that parents are aware these apps are out there and talk to their children as much as possible about their online activities,” she said.
Ms Inman Grant added that work was under way “to protect children from online harms associated with generative AI”.
Do you know more? Confidentially contact marcus.deblonksmith@news.com.au
More Coverage
Add your comment to this story
It is 40 years since we embraced Dire Straits’s Brothers in Arms
Mark Knopfler was Dire Straits: He wrote all the songs, broke pop music’s three-minute rule - and then broke all the records.
Jim Chalmers’ reform cry: Generation Next needs it now
Treasurer Jim Chalmers says the long-term impact of economic reform ‘demands collaboration and compromise’ from politicians, union leaders and business figures.
To join the conversation, please log in. Don't have an account? Register
Join the conversation, you are commenting as Logout