More details about the October 4 outage • Hacker News Board

Re: More details about the October 4 outage

#301

Post by colechristensen » Thu, Oct 07, 2021, 12:48 AM UTC

astrange wrote: post #289

colechristensen wrote: post #148

Earlier quoted context omitted.

Maps, Mail, Drive, Scholar, and Search are all the best or near the best available. That doesn’t mean I like every one of them or I wouldn’t prefer others, but as far as I can tell the competition doesn’t exist that works better. GCP and Pixel phones are a toss-up between them and competitors. It isn’t market dominance, nobody has made anything better.

Search is famously kind of bad the last few years, but even Maps isn’t that great. (Data errors I’ve seen this week: the aerial imagery over Brisbane Australia is from ~2010 but labeled 2021, the coastline near Barentsburg in Svalbard is wrong and doesn’t match any other map.)

I’m not saying any of it is great, just that there aren’t better replacements that make me want to switch.

Re: More details about the October 4 outage

#302

Post by samhwr » Thu, Oct 07, 2021, 8:32 AM UTC

xphos wrote: post #136

I don't know the -f in rm -rf isn't a bug xD. I feel sorry for the poor engineer who fat fingered the command. It definitely highlights an anti-pattern in the command line but the fact that, that singular console had the power to effect the entire network highlights an "interesting" design choice indeed.

It _affected_ the whole network, and _effected_ the whole outage ;)

Re: More details about the October 4 outage

#303

Post by ghostoftiber » Thu, Oct 07, 2021, 3:16 PM UTC

divbzero wrote: post #123

ghostoftiber wrote: post #115

Incidentally the facebook app itself really handled this gracefully. When the app can't connect to facebook, it displays "updates" from a pool of cached content. It looks and feels like facebook is there, but we know it's not. I didn't notice this until the outage and I thought it was neat.

I’m curious if the app handled posts or likes gracefully too. Did it accept and cache the updates until it could reconnect to Facebook servers?

I'm not a huge facebook user and I mostly use it to keep in touch with my parents. I did notice that trying to message them resulted in "message not delivered" (which gave me a prompt to retry) but I didn't try to post anything during the outage.

Re: More details about the October 4 outage

#304

Post by idiotsecant » Thu, Oct 07, 2021, 7:09 PM UTC

mschuster91 wrote: post #11

dmoy wrote: post #4

Interesting bit on recovery w.r.t. the electrical grid > flipping our services back on all at once could potentially cause a new round of crashes due to a surge in traffic. Individual data centers were reporting dips in power usage in the range of tens of megawatts, and suddenly reversing such a dip in power consumption could put everything from electrical systems ... I wish there was a bit more detail in here. What'…

One case is automated protection systems in the grid detecting a sudden hop of current and assuming an isolation failure along the path - basically, not enough current to trip the short-circuit breakers, but enough to raise an alarm.

This isn't really a thing. Transmission and distribution protection doesn't operate on any kind of di\dt basis, other than those defined by overcurrent, in which case the line trips. A sudden increase in load will just manifest in ACE (area control error) as a load imbalance and be dealt with by increasing generation from the spinning reserve that the balancing authority is required to have on hand.

Re: More details about the October 4 outage

#305

Post by oarabbus_ » Fri, Oct 08, 2021, 3:34 AM UTC

Ajedi32 wrote: post #140

oarabbus_ wrote: post #107

Earlier quoted context omitted.

And if the secondary access fails, then what? Backup systems are not reliable 100% of the time.

Then you're SOL. What's your point? The backup might fail, so don't have a backup? I don't understand what you're trying to say.

You should read the post I was responding to and consider the context instead of taking mine in a vacuum. You completely missed the point.

Re: More details about the October 4 outage

#306

Post by charwalker » Wed, Oct 13, 2021, 9:19 PM UTC

BenjiWiebe wrote: post #297

charwalker wrote: post #294

Earlier quoted context omitted.

For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer. Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringi…

Generators do usually start up very quickly. Under a minute.

I guess a DC would have UPS/battery power on hand to cover an instantaneous brown out. Then the generators could be on and restoring battery power while running the DC.