Optus boss waited five hours to call Communications Minister over outage that left 10m Aussies in the dark

Communications Minister Michelle Rowland delivered a blunt message to Optus CEO Kelly Bayer Rosmarin when she finally called her about the unfolding crisis.

Optus chief executive Kelly Bayer Rosmarin says the outage was the result of a ‘cascading failure’ from a ‘network event’ but the same description could be used for Optus’s handling of the crisis.

It took almost five hours after a telecommunications outage that left 10.2 million Australians cut off – with some even unable to call triple-0 – for Optus chief executive Kelly Bayer Rosmarin to pick up the phone and call Communications Minister Michelle Rowland.

The minister had been fielding journalists’ calls from 5am on Wednesday. She completed a radio interview at 7.30am. It wasn’t until 8.30am that her phone lit up with Bayer Rosmarin on the other end of the line, prompting a blunt message from the minister: communicate with your customers.

The fact that Rowland needed to make such a request to Bayer Rosmarin underscores how little Optus has learned from its last reputational disaster when, 18 months ago, hackers released a trove of personal customer data, exposing them to identity theft and other financial crimes.

Then, like now, Bayer Rosmarin and Optus’s response was roundly criticised. Home Affairs Minister Clare O’Neil blasted their management of the cyber ­attack as “not adequate”.

Bayer Rosmarin said the outage was the result of a “cascading failure” from a “network event” but exactly the same description could be used for Optus’s handling of the crisis.

One of its biggest customers told The Weekend Australian that it received a message from Optus about 7.40am on Wednesday, saying it had resolved the problem and its systems were coming back online.

But what it didn’t appear to ­anticipate was a subsequent surge in traffic. Within 10 minutes, Optus’s network had crashed again according to sources – although Bayer Rosmarin denied this version of events.

“Absolutely not,” she said when The Weekend Australian asked her if it was true. “I don’t know where you’re hearing that but I would be curious. What’s your source?”

For almost seven hours following the catastrophic outage, Optus’s public message involved a two-sentence statement.

To fill the communications void, Rowland scheduled a media conference at 11am. Bayer Rosmarin did not think it was necessary for herself to face reporters as a group. Instead, she called ABC radio in Sydney via WhatsApp at 10.30am, and spoke to journalists individually throughout the day.

“We’ve been extremely forthcoming with information and we’ve put a notice out straight away. We’ve kept our messaging very simple and easy to digest,” Bayer Rosmarin said later.

“With customers not being connected to our network, they can’t always receive messages from us,” she added.

Following last year’s falling out with O’Neil, Optus recruited TG Public Affairs, which includes former Labor communications minister, and now registered lobbyist, Stephen Conroy on its advisory board. But this time, they sparked the ire of another minister. Rowland was dealing with the unfolding crisis while performing electorate duties.

While Optus staff were speaking with Rowland‘s office, there was a delay on a direct call between the CEO and the minister.

In her conversation with Bayer Rosmarin, Rowland was forthright in telling the Optus boss that people needed more information. Rowland was frustrated with the amount of information that was coming through.

A government source said: “When things go wrong, the expectation is that you front up. The feeling that is shared throughout the government is that we had to step up and fill that role because Australians had to be reassured that things were OK. We were providing what support we could but the flow of information in those first few hours was critical.”

Despite restoring most of its services by Wednesday afternoon – some customers were still unable to use their phones well into the evening – the government has still not been given the full details of what happened.

“Briefings were happening on Wednesday about what they thought the cause was but there was no definitive 100 per cent ‘this is what happened’,” a government source said on Friday.

“That remains the case.”

After initially saying the cause of the fault was too technical to ­explain, Bayer Rosmarin offered a partial explanation late on Thursday. She said a “technical network event” triggered a “cascading failure” – but she was not able to elaborate further.

“Like most major global telecommunications networks, our network is designed with multiple layers of fallback and redundancy. And at the heart of this is a modern, intelligent router network that we’ve developed with the world’s leading vendors,” she said.

“There’s a misconception out there that the fact we restored the services means we have all the ­answers, and that is not the case.

“Our focus was on restoring services and now our focus is on understanding at the deepest level exactly what happened, in what sequence that allowed a fairly redundant and robust network to not stand up.”

Telco industry insiders say based on these comments, the most likely cause stemmed from a failure in Optus’s border gateway protocol, or BGP.

If you think of your wi-fi router in your house, different devices connect to it, with the internet following around freely. Times that by millions and you get a sense of what a telco network looks like, which involves many routers ­directing traffic to take the most efficient path.

But if one is not configured correctly or updated without another knowing, it can cause them to malfunction, effectively slamming the gates shut across network paths, amplifying or cascading the error like dominoes falling, causing a total shutdown.

A similar BGP failure happened in September 2020, shutting down some of Telstra’s internet services for less than three hours. What makes Optus’s outage more rare is that it shut down its entire network: mobile, fixed line and internet.

Optus now risks losing some of its most lucrative corporate contracts, with major companies from big banks to state governments considering switching to rivals after the meltdown this week.

Since November 2020, Optus has been awarded $547.45m worth of government contracts, according to a website that analyses contracts from AusTender. The outage hit federal departments, including the Department of Foreign Affairs and Trade and the Department of Education.

The Victorian and South Australian governments have said they are both reviewing their contracts with Optus. Victorian Premier Jacinta Allan said Optus’s handling of the crisis was “incredibly disappointing”, with the outage shutting down Metro Trains in Melbourne.

Queensland Deputy Premier Steven Miles said the meltdown was likely to cost businesses millions of dollars.

Optus’s star recruit, former NSW premier and now the telco’s executive in charge of managing its enterprise customers, Gladys Berejiklian, is desperately trying to smooth business relationships after some customers revealed they were in talks with rival telcos.

In an effort to cauterise reputational damage, ­Optus has offered $100 of free data to millions of customers. Optus managing director for customer solutions Matt Williams insisted this was fair, but the offer sparked criticism and anger.

Some small businesses lost as much as $10,000 after the outage crippled their operations, including their ability to accept payments and bookings.

Australian Small Business and Family Enterprise Ombudsman Bruce Billson said the compensation offer was “inadequate” and rejected the telco’s suggestions the loss businesses suffered was $1.

“This is what Optus charges for their services, not what the consequences and loss of that service has cost small businesses in lost income and customers,” he said.

“More needs to be done to acknowledge this impact on the livelihoods of small and family bus­inesses with a tailored response.”