An Update on Our Outage • Hacker News Board

Re: An Update on Our Outage

#231

Post by corpdronejuly » Tue, Nov 02, 2021, 2:49 PM UTC

nonameiguess wrote: post #143

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

Not a software outage, but the Texas ice storm of February resulted in me not receiving mail for two weeks, not being able to use the streets for at least a week (they never plowed and it just eventually melted), rolling blackout where I had power for about four hours a day, and no Internet service at all for the first two days. No water for most of the first day. I'd argue keeping basic city services and roads opera…

Skimping on resilience has been done for decades as part of "cost cutting," in large part because even when we had staffing for resilience, we found that people would collect their check and not actually do the maintenance.

Re: An Update on Our Outage

#232

Post by enjoy-your-stay » Tue, Nov 02, 2021, 8:38 PM UTC

wpietri wrote: post #182

dilyevsky wrote: post #57

Earlier quoted context omitted.

Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code. Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead

So true. And also at the systems level. In most places I've seen, the #1 priority is hitting arbitrary executive feature/date goals, not maintaining robust systems. At some point, the shit will hit the fan, causing a "Why didn't you do perfectly the thing that wasn't a real priority?!?" reaction and a temporary lurch toward robustness. Although often the lurch will be less about actual robustness and more toward perf…

Lol, I can just hear the politicians now.

"Tough on bugs, and tough on the causes of bugs!"

As they add even more time-sapping process to every release.

Re: An Update on Our Outage

#233

Post by withinboredom » Wed, Nov 03, 2021, 11:25 AM UTC

tinyhouse wrote: post #194

withinboredom wrote: post #108

Earlier quoted context omitted.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

From my experience there are a few engineers that are usually key to fixing such issues (not always the same people, depending on the failure). Handing-off work is more for day to day work like handeling a support ticket. So the key engineers need to work crazy hours when things like that happen. But that's OK, ideally these things are rare and people need to work crazy hours only in those rare situations. I'm super…

At Automattic, we handle this by offering a 3-month sabbatical every 5 years and a “minimum” 20 day a year vacation policy. The idea being that no one person should hold knowledge to key systems for too much long.

Re: An Update on Our Outage

#234

Post by redwolfb14 » Wed, Nov 03, 2021, 2:55 PM UTC

jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

This is hilarious. Please add a link to what you implement through k8s secrets, i'd be highly interested in its creation.

Re: An Update on Our Outage

#235

Post by fragmede » Sun, Nov 14, 2021, 9:18 PM UTC

xxs wrote: post #111

colordrops wrote: post #84

Earlier quoted context omitted.

> Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?

the thing can even provide dns... but it has fancy interfaces and an http endpoint. Technically you could do everything with dns just fine. Even if consul provides some sort of liveness (due to latency and concurrency it means little), I can't think of a usable case where the clients won't have retries and ability to maintain multiple open sockets, etc.

Consul's notion of a service includes port numbers, and while I'm sure you could hack something up using DNS txt records and default port numbers, it's an important distinction because it means multiple instances running on one box can easily be discovered.