Optus Senate inquiry as it happened: Kelly Bayer Rosmarin admits telco’s performance during outage was ‘not acceptable’

This concludes our coverage. You can read a full wrap of today’s Senate inquiry in our story here.

Here’s what we learnt today:

• Optus chief executive Kelly Bayer Rosmarin said Optus’ performance “was not acceptable” and the company has “taken immediate and ongoing steps to rectify any shortcomings”.
• More than 200 triple zero emergency calls were unable to go through on the day of the outage.
• Bayer Rosmarin woke up to her phone not working on the day of the outage and rushed to the company’s network operations centre, arriving at the company’s Sydney headquarters about 7.30am, as previously reported by this masthead, and held a crisis meeting between 7.45am until 8.30am at its network operations centre. She said she did not know some customers could not dial triple zero at the time she spoke to Communications Minister Michelle Rowland, which was at 8.32am, about 4½ hours after the outage began about 4.05am.
• Optus is looking at whether it has outsourced too many components of its network infrastructure to third-party companies.
• The telco initially feared the outage may have been the result of a cyberattack, as it was a “very strange coincidence” that the board of its parent company Singtel were in Australia during the network meltdown, just as they were a year ago when Optus suffered a major cyberattack.
• Bayer Rosmarin defended her decision to not hold a press conference on the day of the outage, saying it would have been unusual for a chief executive and that the public’s expectation would have been for her to focus her attention on fixing the problem.
• The CEO said parent company Singtel approved the telecommunication giant’s initial statement that an upgrade at a then-unnamed “international peering network”, later revealed by this masthead to be run by Singtel, contributed to the outage. “The statement that they put out [this week] is not a contradiction, but rather a clarification … They were trying to clarify the way that the statement was being interpreted,” Bayer Rosmarin said.
• Optus did not have a plan for a full outage of the telecommunications network. “We didn’t have a plan in place for that specific scale of outage,” Optus managing director of networks Lambo Kanagaratnam told the Senate inquiry. “It was unexpected.”
• Bayer Rosmarin dodged questions about whether she intended to resign from her role, following a report in the Australian Financial Review on Friday that she may step down as soon as next week. “Senator, I’m sure you can appreciate that my entire focus has been on restoring the outage issue... It has not been a time to be thinking about myself,” Bayer Rosmarin said.
• Optus has paid out $36,000 in compensation for the outage.

• The telco said in its submission to the Senate that while it didn’t wish to detract from the impact of last week’s outage, disruptions were an “inherent risk” and fact of life for telecommunications networks, with a variety of causes to blame, including physical damage, human error, routing errors, software and hardware failures and natural disasters.

Earlier, Optus chief executive Kelly Bayer Rosmarin told the inquiry she had first spoken to Communications Minister Michelle Rowland at 8.32am AEDT – four hours and 27 minutes after the outage began at 4.05am.

A member of her staff contacted the minister just before 8am.

Communications Minister Michelle Rowland.Credit: Alex Ellinghausen

Meanwhile, the telco’s submission to the inquiry reveals Optus had contact with other government ministers and departments about 60 times throughout the November 8 outage.

In addition to speaking to the communications minister, Optus spoke to:

• the office of the minister for communications
• the office of the prime minister
• the deputy prime minister
• the office of the minister for home affairs
• the minister for home affairs
• the office of the attorney-general
• Department of Home Affairs staff
• Department of Communications staff
• each of the offices of the state premier, chief ministers and various state government departments
• the chair of the Australian Communications and Media Authority

“In addition to these individual interactions, the government initiated the National Coordination Mechanism to help update all of government on the status of the outage and to explore opportunities for support,” the submission stated.

“The National Coordination Mechanism organised a meeting at 2.00pm which Optus attended to help provide an update across key Federal and State Government Departments and Agencies.”

Optus has used part of its Senate submission to highlight network outages in Australia and globally, declaring that “each of the Australian telecommunications networks have had outages in the past, some lasting weeks, some lasting hours”.

The telco said that while it didn’t wish to detract from the impact of last week’s outage, disruptions were an “inherent risk” and fact of life for telecommunications networks, with a variety of causes to blame, including physical damage, human error, routing errors, software and hardware failures and natural disasters.

It also said it was giving customers “more than any previous Australian telco that experienced an outage has offered before” with its offers of 200GB in compensation.

“Given continuity of service is not guaranteed, consumers are not given an automatic right of compensation whenever an outage occurs,” the telco said in its submission.

“For example, the Customer Service Guarantee (CSG), which applies to landline services, does not provide for any compensation to be paid to residential or business customers when a service fault is fixed within the day, as occurred in this incident. Similarly, the automatic compensation scheme established by Ofcom in the UK for interruptions to broadband and landline services only provides for compensation where the service is not fixed for two full working days.

“Rather than pay a nominal amount to reimburse customers for the period of time their service was disrupted, Optus has aimed to give customers more than any previous Australian telco that experienced an outage has offered before.

“We understand there are calls by some for Optus to go further, including providing compensation for consequential impacts for business customers. However, there is no precedent for compensation being paid by telecommunications providers to all business customers who suffer a loss of business as a result of an outage of the kind that occurred on November 8, either here or overseas.”

In recent days, Optus and Singtel have been keen to point out the distinction between the trigger event for the outage and its “root cause”

In its submission, Optus attempts to make this clearer, noting the software upgrade on the Singtel Internet Exchange – and subsequent diversion of traffic while it was under way – was a trigger event for the outage. It says the Optus network being unable to handle the significant chunk of new routing information was the root cause of its network being overloaded and crashing.

Optus said its network operations centre observed a loss of connectivity affecting its consumer network about 4.05am on November 8, the day of the incident.

In the initial stages of the outage, Optus said it prioritised the restoration of services as soon as possible, which required re-establishing connectivity to key elements of the network.

“It is now understood that the outage occurred due to approximately 90 PE routers [provider edge routers, which operate between one network service provider’s area and areas administered by other network providers] automatically self-isolating in order to protect themselves from an overload of IP routing information,” the Optus submission says.

“These self-protection limits are default settings provided by the relevant global equipment vendor (Cisco).”

Optus said this “unexpected overload” of routing information occurred after a software upgrade on the Singtel Internet Exchange network, specifically at one of Singtel’s exchanges in North America.

“During the upgrade, the Optus network received changes in routing information from an alternate Singtel peering router, it says.

“These routing changes were propagated through multiple layers of our IP Core network. As a result, at around 4:05am (AEDT), the pre-set safety limits on a significant number of Optus network routers were exceeded. Although the software upgrade resulted in the change in routing information, it was not the cause of the incident.”

Optus said restoration required “a large-scale effort across more than 100 devices in 14 sites nationwide to facilitate the recovery (site by site).

“This recovery was performed remotely and also required physical access to several sites.”

Approximately 150 engineers, technicians and field technicians were in the core group of personnel working on resolution, Optus said.

“That core group was augmented by 250 additional personnel, providing further support and monitoring. We also worked with five leading international vendors who assisted us with resolution and advice.”

Now that they hearing is over, we’re trawling through Optus’ submission to the Senate inquiry.

In it, Optus says there is no precedent to compensate businesses that suffered significant financial losses during the widespread Optus outage.

Optus chief executive Kelly Bayer Rosmarin and managing director Lambo Kanagaratnam appeared before a parliamentary committee on Friday morning to be grilled about last week’s crisis.

The company has been under pressure to pay compensation to businesses after its offer of free data for customers was slammed as inadequate, but Optus has so far refused. In its submission, the telco said no communications network could ever guarantee outages would not occur.

“Reflecting this, communications services are not provided with a guarantee of continuous service,” the submission states. “Consumer and small business consumers are aware that all networks experience outages at some time, and with varying size, severity and time to restoration.

“Given continuity of service is not guaranteed, consumers are not given an automatic right of compensation whenever an outage occurs.”

Bayer Rosmarin told the inquiry Optus was working with small businesses on a case-by-case basis, and so far had paid out $36,000. She insisted the payment was not compensation.

In its submission, Optus states instead of compensating customers $1-$2 for a lost day of service, it decided to go further by offering free data to customers as a goodwill gesture.

“There is no precedent for compensation being paid by telecommunications providers to all business customers who suffer a loss of business as a result of an outage of the kind that occurred on 8 November, either here or overseas,” it said.

“On the question of extending financial compensation for consequential losses, we understand that this would create a new precedent that would extend far beyond Optus and apply to all other telecommunications providers, as well as other providers of essential services, critical infrastructure and public services. This makes it a much broader policy question for government that would have far-reaching implications across many sectors of the economy and the cost of these services for Australian consumers.”

As the hearing draws to a close, Greens Senator Sarah Hanson-Young says she is “flabbergasted” that Optus hadn’t considered letting its customers roam onto the Telstra or TPG networks during the national outage.

“Is it because you don’t want your customers being able to roam on a competitor’s network? Is this about protecting your own profits ahead of the interests of the customer?” she asks. “I don’t understand why this wouldn’t already be on your list of things to investigate.”

Senate inquiry chair Senator Sarah Hanson-Young.Credit: Alex Ellinghausen

Responds Optus chief Kelly Bayer Rosmarin: “Firstly, we don’t see profits and customers as opposed in any way. We only make a profit if we have happy customers, and more customers choose us and more customers choose to stay with us. So for us to be profitable, we want to do the right thing for customers.

“And secondly … given that Optus has ... about 30 per cent market share, let’s say hypothetically that there was a fault on Telstra network, where they have 50 per cent market share, and suddenly all of those subscribers were to be on our network. We would have to have already invested in the capacity to be able to cater for that many customers simultaneously.

“So there are a lot of considerations if we go down this path about investment capacity, and how that all works to make sure that you don’t inadvertently, if one goes down, bring down another network.”

Hanson-Young suggests Bayer Rosmarin “get on with talking to your competitors and making sure the Australian people are actually looked after and can access a service that is essential.”

Nationals Senator Ross Caddell said he had received several emails from Optus staff about poor staff engagement surveys, and then asked Optus chief executive Kelly Bayer Rosmarin whether it was time for new leadership.

“Thank you, senator, I will take that on board,” Bayer Rosmarin responded.

Optus CEO Kelly Bayer Rosmarin said she would take on board feedback about poor staff engagement surveys.Credit: Alex Ellinghausen

Earlier, the chief executive and managing director Lambo Kanagaratnam told senators there was strong feedback from staff on engagement surveys.

Caddell read one of the emails he said he had received from Optus staff: “I just note that engagement surveys we did [over the] last couple of years had nothing about attitude or outsourcing and keeping intellectual property in house. They [the surveys] are pretty much about Optus values. There’s some things about staff development, nothing about workload, how skilled the people are.”

The Nationals senator told the telco chief executive: “Given you haven’t responded to the customers well, given you haven’t reflected your staff’s attitude, isn’t it time for new leadership at Optus?”

Greens Senator Sarah Hanson-Young has asked Optus chief Kelly Bayer Rosmarin whether she has overhauled the company’s PR team following the outage.

“That is one of the biggest problems that members of public have raised with us, and what all of us at this table have put to you today, that the communication with Australians in the midst of this was appalling,” the senator said.

‘We will be taking all of that feedback on board’: Optus CEO Kelly Bayer Rosmarin.Credit: Alex Ellinghausen

“What have you done to fix that?” she asked.

“There are always opportunities for us to do better,” Bayer Rosmarin said.

“As I said in my opening, I wholeheartedly believe that the team did the best that they could with the information they had at the time and the channels available. Having said that, there have been suggestions to us that we do press conferences versus one-on-one interviews, that our messages are a bit different [in] tone, et cetera.

“We will be taking all of that feedback on board to try and make sure that we do better in the future, whilst at the same time trying to make sure there is no future where that’s needed.”

As mentioned earlier, there was a pause (about 3 seconds) when Kelly Bayer Rosmarin was asked several times to respond to a report in The Australian Financial Review that she was considering resigning as the chief executive of Optus following the outage and last year’s cybersecurity breach.

You can replay that tense moment here:

Optus has paid out $36,000 in compensation for last week’s major outage, according to chief executive Kelly Bayer Rosmarin.

She said 8500 customers had sought compensation from the telco giant, totalling $430,000. Optus is in the process of assessing the veracity of those compensation claims, Bayer Rosmarin said.

She added that the telco would be unable to determine the full impact of the outage on small businesses across the country.

Optus CEO Kelly Bayer Rosmarin.Credit: Alex Ellinghausen

The chief executive could not answer whether the $36,000 in compensation was paid out as cash or in-kind services, and told the Senate she would take the question on notice.

When independent Senator David Pocock asked how many small businesses had been affected, Bayer Rosmarin said she had the numbers on her phone. As she took out her phone, senators joked whether her phone was working.

“I’m confident that it’s working,” Bayer Rosmarin responded. When a senator joked she was taking too long to pull up the numbers, the chief executive said: “I think that’s just me. We do have the fastest 5G.”

Optus CEO Kelly Bayer Rosmarin looks through her phone.Credit: Alex Ellinghausen