Less than ‘match fit’ Optus pays a heavy price for failure

The end of Kelly Bayer Rosmarin’s tenure as chief executive at Australia’s second-largest telecommunications company can be essentially traced back to one word. Routine.

At 4.17pm on Monday, November 13, nearly a week after the mass network outage that crippled rail networks, hospital communications and business nationally, an email landed in journalists’ inboxes.

Former Optus CEO Kelly Bayer RosmarinCredit: AFR

Titled ‘Media Alert: Update on Optus outage’ and drafted by Bayer Rosmarin herself, the update was an attempt to shed light on the technical details of the outage after days of virtual silence from both Optus and its parent company, Singtel, as to what had really happened.

“We have been working to understand what caused the outage on Wednesday, and we now know what the cause was and have taken steps to ensure it will not happen again. We apologise sincerely for letting our customers down and the inconvenience it caused,” the email read.

“At around 4.05am Wednesday morning, the Optus network received changes to routing information from an international peering network following a routine software upgrade.”

There was one problem.

Bayer Rosmarin had not used the word “routine”. In fact, Optus’ statement on its own website at the time of publication omits the word “routine”, with the rest of the wording virtually identical.

Executives at Singtel had inserted the word before the statement was sent to journalists.

It may seem innocuous, but Singtel’s intervention goes to the heart of the blame game that unfolded between Optus and its Singaporean parent, and eventually led Bayer Rosmarin, facing an untenable situation on multiple fronts, to fall on her sword.

Optus executive Matt Williams.Credit: Dallas Kilponen

Bayer Rosmarin had already made up her mind to depart Optus’ top job, Optus sources with direct knowledge of the situation said, before fronting the fiery two-hour Senate inquiry that laid bare widespread anger that such a critical service – the nation’s telecommunication’s infrastructure – could go down and for so long.

The Optus outage blocked more than 200 emergency triple-zero calls and shut down rail networks as well as hospital communications services, in a 16-hour event that has been universally deemed unacceptable.

Optus is searching for a new chief executive, with sources suggesting that despite Gladys Berejiklian’s name being bandied about as a hot tip, Singtel’s board will most likely opt for an external replacement, and possibly an executive from another country entirely.

Such a move would give Optus the chance at rebuilding its reputation and communicate to customers – many of whom remain incredibly angry – that the telco has fresh leadership.

The local governance structure Optus has is not designed to help manage it through a crisis.

Ziggy Switkowski, former Telstra and Optus CEO

Alongside Berejiklian, the head of Optus’ consumer division, Matt Williams, has also been touted as a potential CEO, with some of his colleagues praising him as a savvy operator and calm under pressure. He also was present for the data breach, however, and Optus sources say he was directly involved in the decision to offer Optus customers 200 gigabytes in compensation, a move that has been widely panned.

Optus chair Paul O’Sullivan, who does not have a say in the appointment, said in interviews this week the next CEO should be someone with some experience in the telecommunications sector.

A fresh start seems likely.

Ziggy Switkowski holds the unique distinction of being a former chief executive of both Optus and Telstra. “Optus was simply not match fit,” Switkowski said in an interview.

Switkowski, who led Telstra between 1999 and 2004 after leading Optus, knows the ins and outs of a corporate crisis better than most and says Telstra would have handled a similar outage far differently, and likely more successfully.

Ziggy Switkowski says Telstra would have handled the outage differently.Credit: Attila Csaszar

“Telstra has a long history of working ministers and the government, and it also has the responsibilities of dealing with shareholders. It also has a local decision-making structure and a local board. It’s quite clear the line of decision-making and who is accountable.”

“The local governance structure Optus has is not designed to help manage it through a crisis,” he said.

Canberra has made it clear that Bayer Rosmarin’s resignation or even change at Optus’ parent company will mean little compared to ensuring that this month’s outage will never be repeated, a process that will encompass the whole sector and likely mean new rules and regulations.

The appetite for a crackdown – or at least the perception of a crackdown – is strong.

“This was never about which individual is CEO,” Greens senator Sarah Hanson-Young said directly after Bayer Rosmarin’s resignation. “This is about ensuring millions of Australians have access to what is an essential service ... including the ability to call triple zero in an emergency, access government services, contact loved ones, and make and take essential payments.”

Her inquiry is examining stronger regulations for the nation’s telcos, so that, in the words of Hanson-Young, in the event of outages and network failures the community can have confidence that their public interests and safety are protected.

Other inquiries are yet to kick off, including a probe by the communications watchdog into the triple-zero emergency system and a separate inquiry by the department of communications led by minister Michelle Rowland.

Executives at Optus’ rival telcos – Telstra and TPG – have closely watched the events of the past two weeks, and submissions made to last Friday’s Senate inquiry into the outage highlight their anxiety about potential new regulations.

Margins are already anaemic across the sector, with telcos making little from postpaid plans or home internet offerings. Optus itself swung to a $79 million loss in 12 months to March 31, its latest accounts reveal.

Telco executives contacted by this masthead are quick to point out that far from rubbing their hands with glee, as might have been expected, they’re wary about new potential regulations, and the extra spotlight being shone on the industry from Canberra.

They saw how quickly senators from across the political spectrum united in their condemnation of Optus’ handling of its outage, which is a regular occurrence for every telco, though not nearly to that extreme.

Former Optus chief Kelly Bayer Rosmarin and executive Lambo Kanagaratnam face questioning by Senator Hollie Hughes (left) and Senator Sarah Hanson-Young (centre) in Canberra after the outage. Credit: Alex Ellinghausen

“We’re looking now at whether new regulations might lead to higher costs, whether that might have to be passed on to customers, these are the sorts of things we’re talking about,” one executive said, who was speaking on the condition of anonymity to speak freely.

Questions were raised after the Optus outage about why affected customers could not roam on to Telstra or Vodafone’s network instead. The Telecommunications Industry Ombudsman says it has received complaints that customers caught up in the outage were unable to pay for food or bills, make sales or attend medical appointments.

The government is already working with mobile operators to enable temporary roaming during natural disasters, but some industry figures are calling for roaming to be allowed more broadly including during outages.

Telstra, in its submission to the Senate inquiry, warns that it would be too expensive and too risky for it to be forced into sharing its network with rivals in the event of another outage.

“This is because roaming, even temporarily, requires authentication of the end user by their mobile network operator and ultimately the routing of their traffic back to the core network,” Telstra wrote.

“Even if a TDR (temporary disaster roaming) capability, as currently being considered, had been available on November 8, it would not have been able to provide service to Optus’ customers, as all communications with the Optus core network were disrupted.”

Louise Hyland, the chief executive of the Australian Mobile Telecommunications Association (AMTA) – the peak body for the nation’s telco industry - agrees with Telstra’s assessment.

“Roaming to other networks in the event of a major outage poses significant technical and operational challenges,” she said in her submission to the Senate inquiry.

“Most importantly, if the outage disables access to the core of the network then roaming from other networks is not possible as they are unable to authenticate and connect the impacted user devices. Even in situations where roaming is technically possible, there is high risk of the other networks also failing or becoming congested due to the massive surge in the number of devices attempting to connect and communicate with them.”

TPG, on the other hand, says there remains a basis for the government to reconsider mandating domestic roaming, given the benefits it would deliver in rural communities.

“Mandated domestic roaming should be considered as a practical option to bring greater choice and connectivity to regional Australia,” TPG argued in its submission.

“This would also create incentives for increased investment in regional areas and reverse the trend of weakened competitive dynamics. A permanent roaming solution would also have the benefit of providing a much higher degree of network resilience for Australians living in regional areas.”

Meanwhile, The Department of Home Affairs used its submission to argue that the Optus outage highlights the degree to which Australia’s critical infrastructure sectors are deeply interconnected.

Significant disruption in one sector may have severe, cascading and compounding impacts and consequences on the delivery of other critical infrastructure services, the department said.

“The Optus outage has highlighted the consequences which can occur as a result of this interdependency, with the event impacting the availability of other critical infrastructure services, including hospital services, financial services and public transport.

“It also demonstrated the flow-on impact to businesses, which also act as key third-party service providers to other sectors (as well as individuals) across the economy. This adds to the significant impact of the outage on the day-to-day operation of tens of thousands of businesses and millions of individual Australians.”

The department said that longer-term consequences of significant incidents impacting critical infrastructure can often lead to reputational damage and loss of confidence in a system, market, entity or nation – and as a result cause damage to Australia’s national interests.

“The loss of services, regardless of the cause, resulted in significant impacts on a business and personal level, and caused huge consumer distress. The government is considering possible avenues to ensure legislative levers are sufficient to manage these consequences.”

According to Switkowski, Australia’s telcos should co-operate on whatever recommendations come out of inquiries into the outage.

“Some elements of the solution, whether they be too expensive or too commercially risky, have to simply be confronted and accepted. That’s a process and a cost in the modern world, and you can’t avoid it,” he said.

“Sharing access may be necessary, but should be a realisation of the carriers, not imposed by the government. That should be a last-resort action.”

He adds that the lessons out of the Optus incident are clear not just for other telcos, but for the corporate sector more broadly, which has recently grappled with high-profile successive failings from Qantas, PWC, Crown and many others.

“The problem of poor behaviour is not new, there are examples everywhere,” he says. “What is new is the speed and the strength of reaction to them.”

“Given the environment has changed a lot, if you haven’t changed at least as quickly as the environment, or arguably more so, you’re going to be killed. If the world is moving faster than you are, you’re dead.”

The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.