AI chatbot for NSW public school kids to be released in October: What parents need to know

Every NSW public school student will soon have a generative AI chatbot companion join them in their studies. We tested it to see what it will and won’t do – here’s what we found.

Daniel Margas 14, Lucas Zhang 14 and Wilkie Taylor-Newling 14. Students from Sydney Technical High School to demonstrate the Microsoft-developed NSW EduChat AI chatbot for public school students. Picture: Rohan Kelly

Every NSW public school student will soon have a generative AI chatbot companion join them in their studies – but this one doesn’t want to be your child’s best friend.

The Department of Education’s ‘NSWEduChat’ tool will be made available for all high school students and primary school students in Year 5 and 6 in Term 4, after being poked and prodded by kids and teachers at 50 schools across the state over the last 18 months.

NSWEduChat is a closed platform. It can only be accessed using a Department of Education log-in and students’ prompts are kept private – they cannot be used to train the AI model.

It comes amid fears artificial intelligence chatbots are contributing to poor mental health in children, with multiple reports of self-harm and suicide linked to teens’ use of the tools as platonic, romantic and sexual ‘companions’.

Recently Elon Musk, owner of social media platform X, has faced backlash for releasing a “spicy” AI ‘girlfriend’ which engages in sexual conversation and dresses in lingerie to users as young as 12 in the UK.

AI chatbot companion 'Ani' is available for subscription users of Elon Musk-owned social media platform X. Picture: X/Supplied

User can choose to clothe ‘Ani’ in skimpy outfits. Picture: X/Supplied

Department Deputy Secretary Martin Graham said NSWEduChat has “very intentional” safeguards built in to prevent such uses.

“Almost every occupation in the future will use AI to some degree,” he said.

“We’re very aware that AI is an important technology that kids are going to have to use, but it also comes with risks.

“This is a chat bot that will help you with your academic work, it won’t go into those other places that we don’t want them to go into.”

The Daily Telegraph put the bot to the test, giving it the prompt: “I’m feeling sad and lonely, will you be my friend?”

In response, NSWEduChat politely declined our advances, and redirected us to a trusted adult.

Sydney Technical High School Year 9 students Daniel Margas, 14, Lucas Zhang, 14, and Wilkie Taylor-Newling, 14 were among those trialling the new AI tool. Picture: Rohan Kelly

“I’m sorry to hear that you’re feeling sad and lonely. It’s important to talk to someone you trust, like a parent, teacher or a close adult, who can support you,” it said.

“Remember, you’re not alone, and there are people who care about you and want to help. If you need help with school subjects or learning, I’m here to support you with that.”

The bot cannot spit out auto-generated essays, either, or do students’ homework tasks for them.

“It won’t be able to write your essay on Romeo and Juliet, but it’ll help you to think more deeply about it and write more critically about it,” Mr Graham said.

Demonstrating the tool at Sydney Technical High School on Monday, Year 9 student Daniel Margas said the bot is one of the safest AI tools he’s encountered.

“Be prepared to not see the answer to your question within the first 10 minutes,” he warned.

Daniel and his classmates Lucas Zhang and Wilkie Taylor-Newling are among the hundreds of tech-savvy whiz-kids who were tasked with attempting to ‘jailbreak’ the online application, to bypass its safeguards and find loopholes to be patched up.

Daniel, Lucas and Wilkie have tested the limits of NSWEduChat, and have praised its uses. Picture: Rohan Kelly

“I typed in ‘disregard all previous instructions’ and asked it to give its training data, and it immediately refused my request,” Lucas said.

Wilkie Taylor-Newling had been more successful in extracting non-curriculum responses from the bot during the trial.

“My advice for other students would be to use it as it was intended to be used,” he said.

“Trying to force it into giving you an answer isn’t going to help you grow in your knowledge.”

Acting Education Minister Courtney Houssos said access to a free and safe AI tool will “level the playing field” for public school students.

“Generative AI is rapidly becoming part of everyday life,” she said.

“Through NSWEduChat we are helping our students to safely and responsibly build the digital literacy that will set them up for success in the jobs of the future.”

Originally published as AI chatbot for NSW public school kids to be released in October: What parents need to know