Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

151–160 of 235 posts

Re: An Update on Our Outage

#151

Post by AnIdiotOnTheNet »

nonameiguess wrote: post #143
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

Not a software outage, but the Texas ice storm of February resulted in me not receiving mail for two weeks, not being able to use the streets for at least a week (they never plowed and it just eventually melted), rolling blackout where I had power for about four hours a day, and no Internet service at all for the first two days. No water for most of the first day. I'd argue keeping basic city services and roads opera…

That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.

Re: An Update on Our Outage

#152

Post by 0xbadcafebee »

op00to wrote: post #139
xorcist wrote: post #128

Earlier quoted context omitted.

> > We didn’t want to choose any technology that requires the company to drive deep expertise, That's a beautifully concise quote which neatly summarizes what contemporary IT values.

You don’t need deep etcd experience to run k8s. Why is that bad?

You've never had to work a 3 day weekend to recover an etcd failure? Guess I'm just lucky

Re: An Update on Our Outage

#153

Post by bsenftner »

inside_edition wrote: post #55
ZephyrBlu wrote: post #43

Earlier quoted context omitted.

As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?

Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…

This dynamic service terminology sounds very familiar for someone with a few years in the feature film VFX/Animation industries - however, it would not surprise me if Consol/Vault/Nomad were created without a review of similar situations and solutions in other industries.

For those unaware, the feature film VFX/Animation industry has been global and performing large scale technologically ambitious projects requiring multi-company tech collaboration for decades. And not just with code, with assets too: visual assets of all kinds, audio in every possible form, and in some cases legal documents with new approaches to industry issues. Plus these media studios tend to have proprietary workflows, so there is very sophisticated file formats to contain all these information in agnostic manners. All this deep collaboration across creative technical organizations that do not trust one another has developed scalable solutions which the web and formal software development is completely unaware.

Re: An Update on Our Outage

#154

Post by chasd00 »

bsenftner wrote: post #149
keyle wrote: post #82

Earlier quoted context omitted.

In the game industry, working yourself to death for 4 days, that's called a hot fix ;)

And that is after you've been working 6 days a week for the last 9 months. As an EA spouse survivor - fuck that industry.

“Spouse survivor”, I like that! I worked on the tech/distribution/data reporting side of the Covid vaccine and consider my wife, and all the sacrifices she made and extra things she handled up on while I was on the keyboard and phone, a critical component of my team’s success.

Re: An Update on Our Outage

#155

Post by op00to »

0xbadcafebee wrote: post #152
op00to wrote: post #139

Earlier quoted context omitted.

You don’t need deep etcd experience to run k8s. Why is that bad?

You've never had to work a 3 day weekend to recover an etcd failure? Guess I'm just lucky

I’m on the vendor side now, so every day is helping customers recover from etcd failures caused by slow storage due to them not listening to documented requirements. ;)

Re: An Update on Our Outage

#156

Post by bombcar »

blowski wrote: post #124
withinboredom wrote: post #108

Earlier quoted context omitted.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

Automattic always sounds awesome to work for - I know a couple of people who work there as well. In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the o…

Best thing managers can do in these situations is deal with internal politics - be the gateway for information going higher up the chair. The last thing the actual engineering team needs is direct calls from C-level execs.

Re: An Update on Our Outage

#157

Post by darkerside »

ksec wrote: post #138
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

I wonder if Engineers from Roblox are now worth a little more just because they have this experience. I also think it is time that people should take a look again at Chaos Engineering [1] from Netflix. It is sometimes ironic that the best technology often comes from companies that aren't a technology company at all. [1] https://principlesofchaos.org

Are you implying Netflix is not a tech company?

Re: An Update on Our Outage

#158

Post by CrazyStat »

op00to wrote: post #136
withinboredom wrote: post #108

Earlier quoted context omitted.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

This is a benefit of a global company, not working remote. You could do all that stuff in multiple offices around the world.

In a global but not-remote company the team responsible for the particular service that failed is probably still concentrated in one office. Bringing in people from other offices who aren't familiar with the problem service probably isn't that helpful.

Re: An Update on Our Outage

#159

Post by ksec »

darkerside wrote: post #157
ksec wrote: post #138

Earlier quoted context omitted.

I wonder if Engineers from Roblox are now worth a little more just because they have this experience. I also think it is time that people should take a look again at Chaos Engineering [1] from Netflix. It is sometimes ironic that the best technology often comes from companies that aren't a technology company at all. [1] https://principlesofchaos.org

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form.

How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

Re: An Update on Our Outage

#160

Post by NikolaNovak »

blowski wrote: post #124
withinboredom wrote: post #108

Earlier quoted context omitted.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

Automattic always sounds awesome to work for - I know a couple of people who work there as well. In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the o…

One of the greatest values of good technical manager is to run interference/shield for your team. Establish a chain of communication, or if there isn't one in place BE the active reporter, to ensure your hands-on-keyboard people aren't interrupted every 5 minutes (and this is a realistic timeline) for updates from various managers, execs, C-levels, and so on.

Otherwise, it's mostly things you can do before and after an incident like this.

Depending on your role and place in hierarchy, yes DO sit on calls, even if you are quiet; to understand what's going on, to help prioritize if needed, to see if your experienced "spidey sense" catches some risk others may not even though they're closer to the problem (or precisely because of it), encourage, guide, and focus; but all of that without being overbearing or making them feel watched. You need to feel like an ally, part of the team, a friend as opposed to political officer. This again is best achieved ahead of time by demonstrating your value to the team by being supportive and effective always.

Post reply on HN