Upgrade by Optus’s owner Singtel sparked nationwide outage

Optus has confirmed the upgrade that triggered its nationwide meltdown last week was on the network of its parent company, Singtel.

Optus CEO Kelly Bayer Rosmarin initially denied an upgrade was the cause of the outage.

Optus has confirmed the upgrade that triggered its nationwide meltdown last week was on the network of its parent company, Singtel.

The troubled telco — reeling from its second reputational crisis in 13 months after last year’s cyber attack — revealed on Monday that an upgrade on an “international peering partner” sparked the outage, which cut off almost 40 per cent of Australia’s population from phone and internet services last Wednesday.

Some Australians could not even dial Triple-zero for emergency services on fixed lines. Chief executive Kelly Bayer Rosmarin initially denied an upgrade was the outage’s cause, before the telco eventually blamed an international partner.

But on Wednesday morning, Optus confirmed that partner was the Singtel Internet Exchange or STiX, “which sent changes to routing information” shutting down its entire Australian newtork for 11 hours..

“As a rule, we don’t usually name third parties in media responses,” the company said in a brief statement.

Ms Bayer Rosmarin is set to appear before a Senate inquiry into the outage on Friday.

It come as analysts at Maybank — Malaysia’s biggest bank by market value and assets — said Optus could be forced to pay up to $400m under an agreement with Australia’s communications regulator. This would equate to about 10 per cent of its half-year revenue.

The outage shut down Melbourne’s trains ahead of morning peak on Wednesday, the customer contact centre of Australia’s biggest bank CBA and phone lines at some hospitals, among broader disruption.

Singtel’s board - led by Lee Theng Kiat and directors of which include corporate lawyer John Arthur and former Westpac boss Gail Kelly – were also in Australia ahead of the company’s financial results last Thursday. It was the second time the board had all been Australia during an unfolding crisis at Optus. The last time Singtel directors where in the country was in September last year during the cyber attack.

Peering involves two networks exchanging traffic, providing access to one another’s customers.

“The Optus network received changes to routing information from an international peering network following a routine software upgrade,” the company said.

“These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves.”

If you think of your wi-fi router in your house, different devices connect to it, with the internet following around freely. Times that by millions and you get a sense of what a telco network looks like, which involves many routers ­directing traffic to take the most efficient path.

But if one is not configured correctly or updated without another knowing, it can cause them to malfunction, effectively slamming the gates shut across network paths, amplifying or cascading the error like dominoes falling, causing a total shutdown.

Optus said on Monday that now it knows the exact cause, it has “taken steps to ensure it doesn’t happen again”.

“We are committed to learning from what has occurred and continuing to work with our international vendors and partners to increase the resilience of our network. We will also support and will fully co-operate with the reviews being undertaken by the government and the Senate.

“We apologise sincerely for letting our customers down and the inconvenience it caused.”

Optus has also apologised for the length of time it took to restore the network, with some customers still unable to use their mobile phones late on Wednesday evening.

“The restoration required a large-scale effort of the team and in some cases required Optus to reconnect or reboot routers physically, requiring the dispatch of people across a number of sites in Australia. This is why restoration was progressive over the afternoon.

“Given the widespread impact of the outage, investigations into the issue took longer than we would have liked as we examined several different paths to restoration. The restoration of the network was at all times our priority and we subsequently established the cause working together with our partners. We have made changes to the network to address this issue so that it cannot occur again.”

In an effort to cauterise the fallout, Optus has offered customers the equivalent of more than $100 in free data, or 200gb for eligible postpaid customers — an offer that has been roundly criticised after some small businesses lost $10,000 from the outage.

Optus says the offer is fair given the value in lost services would have equated to $1 per customer based on what it charges for its plans. But the small business ombudsman Bruce Billson said this only reflects what Optus charges — not how much customers lost from not being able to access its services.

Originally published as Upgrade by Optus’s owner Singtel sparked nationwide outage