Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More details about the October 4 outage

engineering.fb.com

Post reply on HN
Print view View original

291–300 of 306 posts

Re: More details about the October 4 outage

#291

Post by ethbr0 »

fragmede wrote: post #274
ativzzz wrote: post #250

Earlier quoted context omitted.

Because we as engineers create software for our customers, and if you don't understand who your customers are how can you create software that actually suits their needs? Very rarely are we our own customers

I would argue that SREs are consistently our own customers in a way unique to SRE.

Ironic, as measurability came up in another comment thread I'm in.

I'd say from a technical perspective SREs are, but there's a potential (depends on product) gap between their technical goals and user goals.

e.g. What does "p95 latency is spiking" actually mean to the end user?

Re: More details about the October 4 outage

#292

Post by toast0 »

pm2222 wrote: post #290
toast0 wrote: post #249

Earlier quoted context omitted.

I think this is not necessarily a flaw of the design. It's a fundamental weakness of the real world. Either you can take the backbone being unavailable as a symbol that the PoP is broken, and kill the PoP; or you can take the backbone being unavailable as a symbol that the backbone is broken and do your best. When either interpretation is wrong, you'll need humans to come around and intervene. It's much more common t…

internal tools to bring back the backbone relied on DNS which relied on the backbone being up So are you referring to same DNS servers sitting outside the backbone at the various PoPs? I'd imagine some internal DNS servers which stays in the backbone at use here, unless of course the FB engineers themselves were disconnected from those internal DNS servers.

I don't recall how internal DNS was setup (and determining from the outside isn't really possible), but there were comments in the incident report that DNS being unavailable made it harder to recover.

Re: More details about the October 4 outage

#293

Post by cnst »

mbauman wrote: post #212
cnst wrote: post #210

Earlier quoted context omitted.

An average webpage today is several megabytes in size. How would a few bytes over a couple of UDP packets for DNS have any meaningful impact on anyone's network? If anything, things fail faster, so, there's less data to transmit. For example, I often use ordns.he.net as an open recursive resolver. They use PowerDNS as their software. PowerDNS has the default of packetcache-servfail-ttl of 60s. OTOH, fb.com A response…

The rest of the internet sucked yesterday, and my understanding was it was due to a thundering herd of recursive DNS requests. Slowing down clients seems like a good thing.

You cannot blame other operators if your own operator has broken software.

If your network cannot accommodate another network's DNS servers being unreachable, the problem is your network, not the fact that the other network is unreachable.

A network being unreachable is a normal thing. It has been widely advocated by DJB (http://cr.yp.to/djbdns/third-party.html) and others, since decades ago, that it's pointless and counterproductive for single-site operators to have redundant DNS, so, it's time to fix your software if decades later somehow it still makes the assumption that all DNS is redundant and always available.

I didn't notice any slowdowns on Monday, BTW. I don't quite understand why a well written DNS recursive cache software would even have any, when it's literally just a couple of domains and a few FQDNs that were at stake for this outage. How will such software handle a real outage of a whole backbone with thousands of disjoint nameservers, all with different names and IP addresses?

Re: More details about the October 4 outage

#294

Post by charwalker »

Johnny555 wrote: post #43
plorg wrote: post #17

Earlier quoted context omitted.

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker. A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal t…

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer.

Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringing infra back online in waves to limit spikes.

Re: More details about the October 4 outage

#295

Post by [deleted] »

toast0 wrote: post #125
pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world…

[deleted]

Re: More details about the October 4 outage

#296

Post by jefftant »

toast0 wrote: post #125
pm2222 wrote: post #71

I'm confused by this. Are the DNS servers inside the backbone, or outside? To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection. In the recent outage the entire backbone was removed from operation, making these locations declare themselves unhealthy and withdraw those BGP adve…

If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world…

/* just observing the data presented in the Cloudflare article (https://blog.cloudflare.com/october-2021-facebook-outage/) and disagreeing with the conclusion :-)

While 129.134.30.0/23 (subnet where and a and b nameservers reside) has indeed been withdrawn (according to FB postmortem by DNS automation tooling, 129.134.0.0/17 that is the shorter prefix (perhaps summary at the edge) was still present, however, didn't have longer prefixes (e.g 129.134.30.0/24 and 129.134.31.0/24 we normally see anycasted externally) internally. In other words - routing towards FB DNS subnet (I haven't looked into 185.89.218.0/23 which is where 2 other authoritative nameservers reside) still worked up to the FB border, the traffic was dropped (routed to Null) by FB edge, since it didn't have more specifics internally.

This, combined with TTL of 60 seconds led to almost immediate global DNS failure and all other stuff you have been reading about.

Re: More details about the October 4 outage

#297

Post by BenjiWiebe »

charwalker wrote: post #294
Johnny555 wrote: post #43

Earlier quoted context omitted.

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer. Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringi…

Generators do usually start up very quickly. Under a minute.

Re: More details about the October 4 outage

#298

Post by nomel »

timpattinson wrote: post #238
nomel wrote: post #76

Earlier quoted context omitted.

If your system is pulling 500 watts at 120V, that's around 4A of line voltage. If you drop down 20% to 100V, the output will happily still pull its regulated voltage, but now the line components are seeing ~20% more, at 5A. For brown out, you need to overrate your components, and/or shut everything off if the line voltage goes too low. I used to do electrical compliance testing in a previous life, with brown out test…

"output will happily still pull its regulated voltage" you mean power, right?

All standard computer components require a regulated voltage, then they consume power as a consequence of their operation. The steady voltage is required because the transistors in ICs will break down if voltages go too high, or stop operating if they go too low. Forcing something like an IC to always use the same amount of power, even if it were idle, isn't really possible, because nobody would build it that way.

Re: More details about the October 4 outage

#299

Post by toast0 »

jefftant wrote: post #296
toast0 wrote: post #125

Earlier quoted context omitted.

If we make a simplified network map, FB looks more or less like a bunch of PoPs (points of presence) at major peering points around the world, a backbone network that connects those PoPs to the FB datacenters, and the FB operated datacenters themselves. (The datacenters are generally located a bit farther away from population centers, and therefore peering points, so it's sensible to communicate to the outside world…

/* just observing the data presented in the Cloudflare article ( https://blog.cloudflare.com/october-2021-facebook-outage/ ) and disagreeing with the conclusion :-) While 129.134.30.0/23 (subnet where and a and b nameservers reside) has indeed been withdrawn (according to FB postmortem by DNS automation tooling, 129.134.0.0/17 that is the shorter prefix (perhaps summary at the edge) was still present, however, didn't…

That particular subnet has a covering prefix, but I don't think the other two DNS subnets do, and I had checked on the WhatsApp authoritative subnets, because I have greater affinity for WhatsApp. The WhatsApp subnets don't usually have a covering prefix (and I did check a looking glass during the outage and there were no announcements visible at least at that point).

For those with a covering prefix, the diagnosis is a little bit different as you said, traffic would still flow to whichever FB PoPs advertise the covering prefix, but then it loops in FB, because the PoP doesn't know where to send it, since nowhere was advertising the specific /24. As opposed to the addresses with zero announcements, where the traffic doesn't make it to FB, but gets dropped somewhere else.

Re: More details about the October 4 outage

#300

Post by Johnny555 »

charwalker wrote: post #294
Johnny555 wrote: post #43

Earlier quoted context omitted.

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly. Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer. Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringi…

For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly

If the generators will help in an outage, why wouldn't they help in a brownout? You'd transition to generator when the voltage and/or frequency is outside of spec.

You'd typically have some short-term power protection to keep your datacenter running until the generators start.

I was skeptical about a datacenter that had less than 60 seconds of flywheel energy storage. But the data center manager said that if the generator doesn't start within 30 seconds, you're not going to get it started in an hour so having a huge battery stack that can power the datacenter for 15 minutes isn't going to help much.

Post reply on HN