How One Command Took Facebook Off the Internet for 6 Hours

Just before noon Eastern time on Monday, Oct. 4, 2021, engineers at Cloudflare, a company that runs a big chunk of the internet's plumbing, opened an internal incident with a dull title: "Facebook DNS lookup returning SERVFAIL."

In plain English, their computers had asked where Facebook lived and gotten back a shrug.

The engineers' first worry was that their own system was broken. Then they looked at social media, and the rest of the world had already noticed. Facebook was gone. So was Instagram. So was WhatsApp, which for a lot of the planet is the phone, the group chat and the family dinner table all at once. Cloudflare's Celso Martinho and Tom Strickx later wrote down what went through the room in that first second: "Facebook can't be down, can it?"

It could. And it stayed that way for close to six hours.

What vanished

The scale matters here, so here it is. At the end of September 2021, 3.58 billion people a month used at least one of Facebook's apps, and on an average day that September, 2.81 billion used one. That's a lot of birthday messages, business orders and "are you home yet?" texts that suddenly went nowhere.

People opened Facebook and got "Sorry, something went wrong." Instagram threw up an error message. WhatsApp just stopped connecting.

Downdetector, a site where people report outages, logged more than 10.6 million problem reports by 1 p.m. Pacific time, mostly from the U.S., Germany, the Netherlands and the U.K. By 6:30 p.m. Eastern, the total topped 14 million. It was the largest outage Downdetector had seen up to that point.

Twitter, which was still working, couldn't resist. Its main account posted: "Hello literally everyone."

The strangest part: much of Facebook's equipment was still switched on and working. The rest of the internet just couldn't find it.

How you find a website

To see how a company that size can get lost, you need two pieces of internet plumbing most of us rarely think about.

The first is DNS, short for Domain Name System. Think of it as the internet's address book. You type "facebook.com," which is easy for people. Computers need a number, called an IP address, to actually connect. DNS is the lookup that turns the name into the number. Facebook ran its own DNS servers, the computers that answer "where is facebook.com?" for the whole world.

The second is BGP, short for Border Gateway Protocol. If DNS is the address book, BGP is the road signs. The internet is really a network of networks, run by different companies and stitched together, and BGP is how each network tells the others, "you can reach these addresses through me." Without those signs, there's no route in.

Hold on to both, because on Oct. 4 Facebook managed to lose both at once.

One routine command

Facebook runs what it calls a backbone. It's the company's private network, tens of thousands of miles of fiber-optic cable crossing the globe, linking its data centers. Data centers are the huge warehouses full of computers that actually store your photos and messages. Some hold millions of machines. Smaller sites sit closer to you and connect that private network to the public internet.

The backbone needs constant upkeep. Engineers take pieces of it offline to fix a cable, add capacity or update the software on a router, the traffic-cop machines that decide where data goes next.

That morning, during one of those routine maintenance jobs, someone issued a command meant to check how much backbone capacity was available around the world. Instead, it took down every connection in the backbone. Every data center, cut off from every other.

Facebook had a safety net for exactly this kind of mistake: a tool that checks commands like this one before they run and stops the dangerous ones. That tool had a bug. It let the command through.

The second domino

Here's where a smart design choice made things far worse.

Facebook's DNS servers lived in those smaller sites near the edge of the network. They were built to watch their own health. If a DNS server couldn't reach the main data centers, it would assume something was wrong with its connection and quietly stop advertising itself to the internet. It took down its own road signs.

On a normal bad day, that's sensible. On Oct. 4, every site lost the backbone at the same moment. So every DNS server decided it was the unhealthy one. They all took their signs down together.

Cloudflare watched it happen from the outside. Around 11:40 a.m. Eastern, its systems saw a sudden spike of route changes from Facebook's network, which is usually quiet. Within minutes, the routes to Facebook's DNS servers were gone from the internet's maps.

Put another way: the DNS servers were still switched on and working. Nobody could reach them. Phones and laptops everywhere asking "where is facebook.com?" got the same non-answer.

Locked out of their own house

You'd think the fix was simple. Undo the command.

The problem was that Facebook's engineers used the same network to get into their own systems. With the backbone down, the normal ways in were gone, and so was the backup route, which engineers call out-of-band access. Worse, the loss of DNS broke many of the internal tools the company would normally use to figure out an outage like this one.

So Facebook did the oldest thing in tech support. It sent people to the machines.

That took time, on purpose. Facebook's data centers are built to keep intruders out. They're hard to get into, and even with someone standing in front of the hardware, the routers are designed to be hard to change. Staff had to go through secure access steps before anyone could touch the servers. Only then could they confirm the problem and bring the backbone back.

Facebook's Santosh Janardhan, who wrote up the outage for the company the next day, put the irony in writing: "it was interesting to see how that hardening slowed us down as we tried to recover from an outage caused not by malicious activity, but an error of our own making."

Meanwhile, out in the world

While engineers worked their way in, the rest of the internet got noisy.

Apps don't accept "no" for an answer. When a lookup fails, many try again right away, over and over. People do the same thing, closing and reopening the app, refreshing the page. DNS services around the world were handling 30 times more requests than usual. Lookups for Twitter, Signal and other messaging apps jumped too, as people went looking for somewhere else to talk.

Facebook's statements went out on Twitter, since its own apps were dark. WhatsApp's account posted that the team was "working to get things back to normal."

For people who use WhatsApp as their main way to reach family, the silence was the whole story. The journalist Aura Bogado described it this way: "It's like the equivalent of your phone and the phones of all of your loved ones being turned off without warning."

Turning the lights back on

Around 5 p.m. Eastern, Cloudflare saw Facebook's network start sending out route updates again. By about 5:20 p.m., facebook.com was answering lookups once more.

Here's the part almost nobody saw. Getting the backbone back was not the same as getting the apps back.

While the services were down, individual Facebook data centers had dropped their power use by tens of megawatts, an enormous amount of electricity. Flip everything back on at the same moment and that load would slam back in, with a flood of returning users right behind it. That surge could knock things over all over again, from the electrical systems to the caches, the short-term memory that keeps apps fast.

So they brought it back slowly. Facebook had practice. For a long time it had run what it calls "storm" drills, where it deliberately takes a service, a data center or a whole region offline to see what breaks. That experience let the team ramp the load up carefully. Some people got partial access around 5:45 p.m. Eastern. Shortly after 6:30 p.m., the company posted on Twitter: "To the huge community of people and businesses around the world who depend on us: we're sorry."

Mark Zuckerberg posted his own apology: "I know how much you rely on our services to stay connected with the people you care about."

And this wasn't an attack. There was no malicious activity and no evidence that user data was compromised. In Facebook's own words, the root cause was "a faulty configuration change on our end." A settings change, made by Facebook, to Facebook.

What it shows about your phone

None of this is something you can fix from your couch. But it does explain an experience you've had.

When one app shows an error and the rest of your phone works fine, the problem may not be on your end at all. That afternoon, anyone who restarted a phone or reset a router to fix Instagram was fixing something that wasn't broken. A quick look at a site like Downdetector, or at a news site, could have answered the question. And hammering refresh doesn't help. That 30-times jump in lookups is what it looks like: more traffic piled onto a system that's already struggling.

It's also a decent reason to keep a second way to reach the people who matter most. A plain phone number, a regular text or an email address doesn't depend on any one app.

The drill they hadn't run

The storm drills are the detail that lingers. Facebook had spent a long time practicing for disaster, knocking out whole data centers and whole regions on purpose, so that on a bad day the team would know what to do.

It had drilled for a lot. Janardhan admitted a gap in his write-up: "we've never previously run a storm that simulated our global backbone being taken offline."

The disaster they hadn't rehearsed was the one where the backbone itself went dark, and every road sign came down with it.

The Day Facebook, Instagram and WhatsApp Fell Off the Internet