An Update on Our Outage • Hacker News Board

Re: An Update on Our Outage

#131

Post by mianos » Mon, Nov 01, 2021, 10:23 AM UTC

dilyevsky wrote: post #59

ZephyrBlu wrote: post #54

Earlier quoted context omitted.

So relating to things I do know about, consul is like cloud infra, vault is a service for secret storage and nomad is like Kubernetes?

Consul is like etcd but has some extra features built in like service discovery and l7 proxy so they market it as full blown service mesh a la istio, the other two are spot on

I think this is more correct. Like Etcd in that it is a distributed kv store with consensus. It has a dns interface that facilitates service discovery but it can be used for so much more. Infrastructure like global locks, coordination or even just a fault tolerant kv database, as it is for vault itself.

Re: An Update on Our Outage

#132

Post by raxxorrax » Mon, Nov 01, 2021, 10:36 AM UTC

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

We have hospitals being knocked out for weeks after ransomware attacks. If we complain about roblox I think we should reflect on our priorities.

Re: An Update on Our Outage

#133

Post by Railworks2 » Mon, Nov 01, 2021, 10:48 AM UTC

celim307 wrote: post #20

Bad timing too, I bet holiday events like Halloween are a big deal in the roblox culture

Roblox Dev here (Make the experiences, not employee), Halloween the second most important weekend for us developers in terms of income.

The Friday and Saturday are critical days in getting revenue from any Halloween specific events we may do. Such as a Halloween update or discount.

I'm happy to see Roblox is refunding adverts, especially after seeing how much support this has. https://devforum.roblox.com/t/reimbursement-on-sponsors-and-...

Re: An Update on Our Outage

#134

Post by buryat » Mon, Nov 01, 2021, 10:52 AM UTC

jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

you can simply disable parts of functionality if it's truly consul that's responsible

Re: An Update on Our Outage

#135

Post by Railworks2 » Mon, Nov 01, 2021, 10:52 AM UTC

mmmBacon wrote: post #79

_nickwhite wrote: post #9

My 3 youngest kids, aged 9, 9, and 8 told me Roblix were having an extended outage, world wide. Good job to the team getting everything back up. Been there, done that. #hugops

My 8 y.o. daughter ran through the house with an excitement typically I only see on Christmas morning exclaiming “Roblox is back!”

Feels like that for me, suddenly I can actually reopen Roblox Studio and do some work.

Re: An Update on Our Outage

#136

Post by op00to » Mon, Nov 01, 2021, 10:54 AM UTC

withinboredom wrote: post #108

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

This is a benefit of a global company, not working remote. You could do all that stuff in multiple offices around the world.

Re: An Update on Our Outage

#137

Post by yawgmoth » Mon, Nov 01, 2021, 10:54 AM UTC

Cthulhu_ wrote: post #117

nodejs_rulez_1 wrote: post #116

Earlier quoted context omitted.

I am sure they got a lot of stock.

I mean I'd gladly pull an all-nighter if I were a millionaire thanks to the thing I'm fixing. I think. Then I'd quit and do something leisurely.

Over what time frame would you tolerate poor working conditions to make a million bucks? Four year equity vesting? A million dollars in 2025 too, not a million dollars in 2010.

Re: An Update on Our Outage

#138

Post by ksec » Mon, Nov 01, 2021, 10:55 AM UTC

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

I wonder if Engineers from Roblox are now worth a little more just because they have this experience.

I also think it is time that people should take a look again at Chaos Engineering [1] from Netflix. It is sometimes ironic that the best technology often comes from companies that aren't a technology company at all.

[1] https://principlesofchaos.org

Re: An Update on Our Outage

#139

Post by op00to » Mon, Nov 01, 2021, 10:56 AM UTC

xorcist wrote: post #128

terom wrote: post #101

Earlier quoted context omitted.

Choice quotes from their PR piece: https://www.hashicorp.com/case-studies/roblox > We didn’t want to choose any technology that requires the company to drive deep expertise, almost to the point where you have to be a code contributor back into the project to get what you want. Nomad is just very easy to adopt. Better be damn sure you have your 24/7 vendor support contracts in order if and when shit does hit the fan.

> > We didn’t want to choose any technology that requires the company to drive deep expertise, That's a beautifully concise quote which neatly summarizes what contemporary IT values.

You don’t need deep etcd experience to run k8s. Why is that bad?

Re: An Update on Our Outage

#140

Post by withinboredom » Mon, Nov 01, 2021, 11:10 AM UTC

blowski wrote: post #124

withinboredom wrote: post #108

Earlier quoted context omitted.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

Automattic always sounds awesome to work for - I know a couple of people who work there as well. In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the o…

There's avid communication and coordination is all done via text (Slack/IRC). As long as you're following along, I think you can spot it pretty well. People are also more used to managing their own stress and time. They will let the people know if they need to step out for a few minutes so the workload could be shifted/paused to give them time to step away.