Facebook outage a nightmare

Australians woke yesterday to Facebook and its bevy of apps crippled by one of the company’s worst global outages in its history.

Former Facebook employee Frances Haugen’s appearance on US TV preceded the more than six hour Facebook outage. The two events created a nightmare for Facebook.

Australians woke yesterday to Facebook and its bevy of apps crippled by one of the company’s worst global outages in its history. Facebook, Messenger, Instagram, and WhatsApp went offline some time after 2am.

People from Canberra, the Gold Coast, Brisbane, Townsville, Palm Island, Adelaide, Melbourne, the NSW Central Coast, Perth and Sydney were among hundreds who used alternative social media to complain.

Services didn’t start to restore until after 9am, just as Australians began their day jobs. Some users still experienced issues around lunchtime.

The outage was part of a nightmare two days for Facebook. In the US viewers on Sunday night watched an American 60 Minutes interview with Facebook Whistleblower Frances Haugen, a former product manager, who revealed the company knew of the hate, violence and misinformation being spread on its platform but didn’t care. Profits came first.

The Wall Street Journal had recently published Haugen’s claims in a sensational series of stories called The Facebook Files which were now centrestage across the nation.

The outage brought further humiliation. Some Facebook employees and contractors tasked with fixing service issues couldn’t log into the Facebook system and access their software tools to fix the problems. There were reports that Facebook staff couldn’t even enter the building.

A major cyberattack was discounted. Denial-of-service attacks seemed unlikely due to the sheer size of Facebook‘s online footprint, internal sabotage was raised as a possibility, but the outage was eventually blamed on misconfigurations of Facebook’s DNS (domain name system).

It means the links between Facebook and its allied services links to the internet pointed to the wrong addresses.

The Facebook outage as graphed by monitoring site downdetector.com.au.

Facebook has offered no reasons for the outage, but CEO Mark Zuckerberg did apologise.

“Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today — I know how much you rely on our services to stay connected with the people you care about.”

Robert Potter, CEO of Security firm Internet 2.0, said if Facebook got its internet addressing wrong, it could “crash a lot of systems”. He says it is true you may lose your ability to fix things at the same time.

“There‘s certainly no evidence of a cyberattack of that scale forthcoming yet.”

Mr Potter said he believed Facebook’s IT operations team could soon receive “a strongly worded letter explaining to them exactly how many dollars they lose per minute”. One estimate puts the revenue loss at $US60m. In addition, Facebook’s share price dropped almost five per cent in a day.

The outage started to end just after 9am, when Australians were preparing to start their day jobs. The impact could have been worse if it had continued.

But it did again prompt renewed comment about the dependence of Australian business on Facebook, an issue sheeted home when Facebook blocked Australian users from viewing or sharing news in February.

Facebook later published an engineering blogpost blaming “configuration changes” for the outage. “This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.” The post said there was no evidence user data was compromised.

Macquarie Telecom senior Adviser Marcus Thompson, a retired Army officer and former ADF Head of Information Warfare, said thousands of Australian organisations, commercial enterprises and not-for-profits rely on Facebook and had woken up with no alternative platform to reach their stakeholders.

“The outage highlights the over-reliance Australian businesses have on international digital platforms and systems. When cyberattacks or other disruptions inevitably strike these international platforms, with no local, sovereign networks, infrastructure or methods to fall back on, we’re at the whim of clean-up and remedial measures taking place half a world away.”