Facebook, Gmail outages give us a warning

Google was down and Facebook has had a record outage. There’s a warning we can’t ignore.

Facebook CEO Mark Zuckerberg. Picture: AFP

Facebook is coming back up after a record outage of up to 11 hours in some areas and many people are wondering how this happened.

Facebook says the outage was not caused by a “DDOS” (distributed denial-of-service) attack, where hackers bombard a service with enough traffic to cause it to fall over.

A software update that went horribly wrong could be a possibility, but given that Facebook, Instagram, Messenger and WhatsApp were all affected, it appears to be related to the networks that distribute Facebook’s traffic around the world.

It would be terribly hard for Facebook to lose your data. Facebook keeps copies of user data at several places around the world so that it can respond to outages, as well as deal with the possibility that its data could be destroyed at any one of these locations by war or natural disaster.

Even back at a data conference I attended in Santa Clara some years ago, Facebook acknowledged keeping a complete replica of user data in four data centre regions around the world, and was adding a fifth region at the time.

The outage might be a minor annoyance for some who use Facebook or Instagram recreationally, but for organisations that depend on it professionally, today is a disaster.

Advertising trade publications already are reporting the effect it is having on ad serving and revenue.

We know. Our @facebook and @instagram haven’t been working either. Unfortunately we cannot do anything about this because, you know, they’re based in America and we’re the Police. So please don’t call us to report this. Pretty please.

— Canterbury Police NZ (@NZPCanterbury) March 13, 2019

Organisations that depend on Facebook to reach people in their communities also will be disadvantaged.

In New Zealand, police today urged the public not to call its 111 emergency services hotline to report that Facebook is down. “Pretty please,” they asked.

In the end, an outage of 11 hours-plus has shown how dependent the world has become on social media and is a wake-up call.

The outage comes a day after Google’s Gmail service also went down.

Hopefully both will explain in some detail what happened.

In the meantime, a University of Sydney expert in online systems has a theory as to what happened. We don’t know what happened officially but he believes today’s outage was caused by “border gateway protocol (BGP) hijacking”, also known as prefix hijacking and route hijacking.

It’s when the internet no longer knows how to route some data because the destination address is corrupted. Imagine, for example, if the table of postcodes (zipcodes) was destroyed, so that the post office didn’t know where to send a letter labelled with postcode 2000.

Instead of postcodes, the internet uses groups of IP addresses to decide how data is routed to its destination.

Professor Albert Zomaya, Director for Distributed and High Performance Computing at the University of Sydney School of Computer Science, says the outage may or may not be malicious.

“The Internet enables any connected device with a unique IP address to talk to any other device. These IP addresses are used to construct routing tables which tell a packet how to get to its final destination,” Professor Zomaya tells The Australian.

“Globally, each group of IP addresses are grouped together into prefixes that are owned by an organisation, such as Google, Facebook, or a university. Each organisation is known as an autonomous system.”

Massive routing problems occur if these tables of prefixes are hijacked or corrupted.

“This happens when a group of IP addresses are taken over by corrupting the routing tables that are normally maintained by the BGP,” Professor Zomaya says.

“When this happens packets will not be properly delivered and will cause serious outages or even a complete loss of connectivity.”

This corruption could be innocent but if it’s malicious – and there’s a chance it is – it could compromise users’ security.

He says BGP hijacking can be used by malicious users to obtain IP addresses for use in spamming or distributed denial-of-service attack.