Facebook’s platforms behave like old Christmas lights

Facebook’s various social media platforms are like old Christmas lights. When one goes out, they all go out. Why did Facebook, Instagram, Messenger, WhatsApp and even Oculus fall over like cards?

That’s an issue which has been exercising the brainpower of business, strategy and operations support firm Forrester.

Its blogpost today suggests Facebook might have tripped itself up trying to beat the regulators.

Forrester said that over the past two years, Facebook had consolidated its disparate app ecosystems onto one back-end infrastructure. No longer were Facebook, Instagram and the others separate, they were part of one system.

This could have been achieved for several reasons. Forrester says that on the one hand, the move created some operational efficiencies for the company. The apps do cross-reference user data and all gain insights into user behaviours. There is good reason to do such a thing.

On the other, the Forrester blogpost says the move has helped insulate the company from a potential breakup by regulators. Forrester‘s argument suggests Facebook would be safer from being broken up if its various platforms were intertwined.

The move exposes Facebook to concentration risk, says company analysts Mike Proulx, Jessica Liu, Jeff Pollard and Alla Valente in their blogpost.

They said the move created a single risk event that produced a cascading effect “like old school Christmas lights where one goes out, they all go out”.

“This strategy comes at the expense of redundancy and impairs the company’s resilience. It also irritates consumers who don’t want a unified social media profile across Facebook’s family of apps.”

They said the outage showed that Facebook has a risk-management problem.

“Although many brands had a ‘real-time marketing’ moment on Twitter today, this outage has widespread implications to the advertising ecosystem given the fact that ads weren’t being served for over six hours across Facebook and Instagram, which command the lion’s share of social media ad revenue.

“This not only affects Facebook’s revenue (and stock price) but also brands’ bottom lines.

The explanation, which Facebook has yet to offer, may be more complex or it may come down to a simple issue – that the internet could no longer find Facebook, Instagram, Messenger, WhatsApp and Oculus VR because someone had changed their online addresses.

In olden days, when the postie couldn’t deliver mail to an address, it was readdressed “return to sender”. In the digital 21st century, you just get a blank screen or error message.

In an analysis piece today, Doug Madory, director of internet analysis at Kentik, argues that not only did the internet lose track of where Facebook actually is, IT engineers couldn’t access software tools to fix it because the internet addresses used to access them also no longer worked.

Facebook has left the car running full bore in gear with the keys locked inside.

“In simpler terms, sometime this morning Facebook took away the map telling the world’s computers how to find its various online properties,” argues Madory on security specialist site KrebsonSecurity. “As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com, and so returns an error page.

“In addition to stranding billions of users, the Facebook outage also has stranded its employees from communicating with one another using their internal Facebook tools. That’s because Facebook’s email and tools are all managed in house and via the same domains that are now stranded.

“We don’t know how or why the outages persist at Facebook and its other properties, but the changes had to have come from inside the company, as Facebook manages those records internally. Whether the changes were made maliciously or by accident is anyone’s guess at this point.”

In addition, there had been a report that the misconfiguration meant that staff entry badges into buildings no longer worked. People couldn’t get inside to mend things either.

Either way, it seems the “old Christmas lights” analogy does describe the modus operandi of Facebook and its linked platforms.

Facebook CEO Mark Zuckerberg has issued a brief apology. “Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today -- I know how much you rely on our services to stay connected with the people you care about.”

UPDATE: Facebook has now published an engineering blogpost blaming configuration changes for the outage, as we thought.

“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centres caused issues that interrupted this communication,” the engineers post says. “This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.

“Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.”