Optus outage: The ‘very technical’ issue that took down a major telco

Optus CEO Kelly Bayer Rosmarin has shut down rumours an overnight upgrade took down the telco and apologised to customers affected by its outage.

Optus chief executive Kelly Bayer Rosmarin has defended the company’s response to Wednesday’s major network outage in the face of strong criticism from angry consumers unable to make phone calls or send or receive messages.

“Of course, we’ve been extremely forthcoming with information and we’ve put a notice out straightaway. We’ve kept our messaging very simple and easy to digest,” she said.

“But with customers not being connected to our network, they can’t always receive messages from us.”

Ms Bayer Rosmarin late on Wednesday told The Australian that Optus had fully restored its network and the company was identifying the root cause of the outage, which took down the network for the majority of its 10.2 million customers.

Optus “had some ideas” what had caused the outage but would wait until its engineers had conducted a full investigation before sharing further details, she said.

“What I can say is it is a very technical network engineering issue.”

Ms Bayer Rosmarin denied the suggestion the outage had arrived after a major network upgrade overnight. “I can tell you that is not true,” she said.

Earlier on Wednesday, telco industry insiders had told The Australian that a routine upgrade, which if all had gone according to plan, would have taken place without detection in the small hours of the night, was behind the service going down at 4am.

No services were spared amid the outage, not even that of the Optus chief executive Ms Bayer Rosmarin, who was left to dial into major interviews via the Meta-owned messaging service WhatsApp.

Optus CEO Kelly Bayer Rosmarin.

The Australian was told from industry insiders that the upgrade had disrupted Optus’ central nervous system and that key components of the network had needed to be reset to get the network back up and running.

Most telcos don’t give the public too much insight into their network architecture, with a black box-like approach to deter hackers and competitors from copying or stealing models.

The resetting of key network components was explained as the reason for why some customers had experienced ­different access to services such as some being able to send text messages, while others were limited to internet access.

Throughout Wednesday, some customers had declared they would leave the telco following a lack of communication from Optus about the outage.

To that, Ms Bayer Rosmarin said: “I would say to those customers that we are a company who really champions the customer, and we work very hard to give our customers great value for money, and a very reliable and excellent network and unique experiences that they can‘t get anywhere else.

“And on almost every other day we succeed in delivering on that promise for our customers. And today we’re very sorry that we let them down but that is a very rare occurrence.”

Asked if Optus would provide more details to customers in the coming days, Ms Bayer Rosmarin said what most customers wanted was to be connected and they now were and “hopefully they’ll get back to getting on with their daily lives, their businesses and enjoying our great quality network”.

Ms Bayer Rosmarin ruled out holding a press conference in coming days, adding “I really don’t think that that’s necessary. But I’m open to any views”.

“We’ve communicated very openly and transparently. When we have a full root cause analysis, if it’s relevant, we will share that and be forthcoming,” she said.

Internet access is needed for train networks to function and to communicate from station to station as well as from train to headquarters, as was the case in Melbourne with its Metro Train coming to a halt on Wednesday

Melbourne Metro uses high capacity signalling to automatically adjust the speed of trains to maintain safe distances between nearby trains.

Customers leaving an Optus store in the Sydney CBD on George St on Wednesday. Picture: NCA Newswire /Gaye Gerard

Technologies such as IoT (Internet of Things) are used to monitor temperature, provide location and determine accidents on transport networks.

The outage and its impact was considered so severe the South Australian government engaged its State Emergency Centre, with Acting Police Commissioner Linda Williams having confirmed the number of calls to triple zero had risen following the outage.

The reason for that, she said, was because phone lines into SA Health had dropped in the outage, with patients unable to reach hospitals in SA Health’s network. Inside those facilities, doctors and clinicians were unable to communicate internally over the phone or by text and instead were instructed to use applications including Microsoft Teams.

With devastating bushfires forecast throughout the week, Commissioner Williams expressed major concerns over Alert SA’s ability to reach customers and notify them when they need to leave their homes or get out of the way of a blaze.

“Alert SA, which is the service that notifies people of emergencies, has been impacted by the outage,” she said. “It is particularly important heading towards the end of the week where we see elevated fire danger for parts of the states.”

SA Premier Peter Malinauskas confirmed his government was considering switching telcos following the outage, and added that: “We are actively engaging with Telstra as the Deputy Commissioner referred to regarding what we can do to switch over some of the critical services that we want to resolve.”

Originally published as Optus outage: The ‘very technical’ issue that took down a major telco