‘We’re not to blame for outage’: Singtel fires back at Optus

The Singapore government-controlled telco says it’s not to blame for the mass outage after Optus claimed an upgrade on Singtel’s network led to a ‘cascading failure’.

Singtel says a routine upgrade on its network was not the root cause of Optus’s national meltdown that cut off 10 million Australians last week.

Optus chief executive Kelly Bayer Rosmarin initially denied that an upgrade triggered the outage. But on Wednesday, Optus finally revealed an upgrade from its owner, Singtel sparked the shutdown.

The comments prompted Singtel to issue a statement of its own on Thursday. The Singapore government-controlled telco said Optus – which is reeling from its second reputation crisis in 13 months after last year’s cyber attack – was well aware that it was running a “routine upgrade” on the Singtel internet exchange known as STiX.

It also said after it completed the upgrade – which took about 20 minutes – Optus’s network was “up and running”. Optus later clarified that it agreed the fault was in its Australian network.

It comes on the eve of Ms Bayer Rosmarin appearing before a Senate inquiry into the outage.

“We are aware that Optus experienced a network outage after the upgrade when a significant increase in addresses being propagated through their network triggered preset fail safes. However, the upgrade was not the root cause,” a Singtel spokesman said in response to questions from The Australian.

“From time to time, STiX conducts software upgrades for its infrastructure. STiX had planned for a routine software upgrade on one of its routers at 1am SGT (Singapore time) on November 8 2023, and as per usual practice, gave prior notice to all affected customers, including Optus and other telecom companies.

“During the upgrade, data traffic was routed to other points of presence on the STiX network, and back into customers’ networks. The STiX upgrade was completed within 20 minutes, and all its customers’ routers that were connected to it, including Optus’, were up and running.”

Singtel said it “will support Optus as it learns from what has occurred and continues to improve”.

“Network resilience and security remains a top priority of the Singtel Group”.

An Optus spokeswoman said later that: “Singtel’s position on the root cause is not contrary to ours”.

“In our earlier statements, we had clarified that the root cause was not the external software upgrade but an increase in addresses being propagated through our network that triggered a failsafe mechanism.

“The affected routers then disconnected from the Optus IP Core network to protect themselves. We are making changes to our network to ensure it will not happen again.”

Earlier this week Optus finally revealed that a network upgrade triggered the outage, that shut down Melbourne’s train network, the call centre of Australia’s biggest bank CBA and communications at some hospitals among broader economic disruption.

But Optus was at pains to highlight that the upgrade was on an international network – not its own – before exposing Singtel as that overseas provider on Wednesday.

“As a rule, we don’t usually name third parties in media responses,” Optus said.

In a private call with Singtel investors last Thursday, dismissed the prospect that Optus could face penalties for the outage that also impacted the Triple-zero emergency service.

“Of course nobody has an outage like that on purpose, so I think it’s just premature to even speculate in that direction,” Ms Bayer Rosmarin said, according to a recording of the briefing.

She also knocked back suggestions customers should get cash returns as compensation for the crash.

“I just want to explain that we’re not talking about direct compensation because if you took an average customer on a $49 plan, and they didn’t have service for a day, that would equate to about $1.60, and we don’t think that’s what customers want, is a credit for $1.60”.

“We’re looking at what we can do to make our customers feel like they’ve been heard, that they know that we care, that we understand that we let them down and to try and give them something more valuable.”

Analysts at pan-Asian investment bank Maybank said Optus could be forced to pay up to $400m under an agreement with Australia’s communications regulator. This would equate to about 10 per cent of its half-year revenue.

Maybank based its calculations on the compensation Telstra paid customers following an outage in 2016, which involved a $25 credit for consumers and $50 credit for businesses.

Optus has since offered customers the equivalent of more than $100 in free data, or 200GB for eligible postpaid customers.

The blame games between Optus and Singtel are in sharp contrast to the leadership style of Telstra CEO Vicki Brady. Brady said she takes no pleasure in seeing a rival suffer, but she is taking lessons from them to strengthen Telstra’s own network.

“No one is immune from outages,” Brady said on Wednesday. Telstra has a shut down in 2016.

“What is fundamental to us, though, is absolutely designing in the resilience, making sure as any big outages happen around the world, we make sure we understand the learnings from that and what do we need to think about building. It’s been a core focus of Telstra for a long time.”

Specifically, Brady talks about creating “blast zones” to ensure that when things do go awry, it doesn’t shut down the entire network.

“Obviously the worst is national – having issues that are going to take you down nationally. So as we design the network and think about where there’s potential points where things could fail, we think about how do we contain that to a state or to a geographic area or to a certain part of the service we’re providing.”.