Hello, welcome to the Tuesday, October 5, 2021 edition of the Sands and the Storm Center's Stormcast.

My name is Johannes Ulrich.

And I'm recording from Jacksonville, Florida.

Big story today, of course, was Facebook's outage.

And while, of course, it's not necessarily sort of a security event that

affects many of us, has made the news big time. So I figured I'll talk a little bit about

what we know has happened at this point. As I'm recording this, Facebook is a backup now for

about half an hour or an hour after being down for

approximately five and a half or six hours. There is no official statement yet from Facebook

as to what happened, but Brian Kreps is reporting and he usually is right about these things that the root cause was a BGP update that apparently went wrong.

Now initially, a lot of reports suggest that it's a DNS problem.

And of course, DNS is a common culprit for issues like this, and DNS certainly had

issues, but DNS was only down because, well, there was a BGP, a routing problem. At approximately

1130 Eastern or 1530 UTC, the Facebook IP address prefixes were withdrawn from Global Routing

Table, pretty much bringing down anything Facebook related.

So Facebook, Instagram, as well as WhatsApp, were not reachable because while the Internet

no longer knew how to reach the respective

IP addresses. And with that, of course, Facebook DNS servers were also not reachable,

and that's why it sort of manifested itself as a DNS problem initially. What prolonged the

outage was that the BGP update not only removed access

from the outside to Facebook, but also router administrators were no longer able to actually

reach affected routers in order to fix the problem that the initial bad update caused.

...