Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

101–110 of 235 posts

Re: An Update on Our Outage

#101

Post by terom »

inside_edition wrote: post #41
lwansbrough wrote: post #31

Someone on here posted the other day that someone they knew at Roblox said it was their secret store that became overloaded. Presumably a Hashicorp Vault type service (or something similar.) This update appears to support that claim. That's definitely a problem you only get at scale.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…

Choice quotes from their PR piece: https://www.hashicorp.com/case-studies/roblox

> We didn’t want to choose any technology that requires the company to drive deep expertise, almost to the point where you have to be a code contributor back into the project to get what you want. Nomad is just very easy to adopt.

Better be damn sure you have your 24/7 vendor support contracts in order if and when shit does hit the fan.

Re: An Update on Our Outage

#102

Post by Ueland »

mr-karan wrote: post #4

Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…

I just refreshed and it worked the second time.

Re: An Update on Our Outage

#103

Post by faizshah »

jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

Just curious, what are the common ZK failure modes?

Re: An Update on Our Outage

#104

Post by lmm »

colordrops wrote: post #84
inside_edition wrote: post #55

Earlier quoted context omitted.

Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…

> Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?

Dynamic-first, better support for failover etc.. It's probably not doing anything that you couldn't build on top of DNS, but it's designed from the ground up for clustered deployments.

Re: An Update on Our Outage

#105

Post by zeeZ »

anilgulecha wrote: post #90
colordrops wrote: post #84

Earlier quoted context omitted.

> Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?

Really high level, it doesn't. It's an internal dynamic DNS, essentially.

One of their primary query interfaces actually even is DNS.

Re: An Update on Our Outage

#107

Post by strictfp »

jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

My stance is: why even have any of those? Are you running in a local datacenter?

Re: An Update on Our Outage

#108

Post by withinboredom »

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

Re: An Update on Our Outage

#109

Post by withinboredom »

colordrops wrote: post #84
inside_edition wrote: post #55

Earlier quoted context omitted.

Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…

> Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?

DNS gets cached all up and down the stack. Your OS, routers, application libraries and probably even your application code might cache it. Consul allows you to have it handle the caching, and be notified when a service goes away so it won’t return that address to your application.

Re: An Update on Our Outage

#110

Post by donkeyd »

sverhagen wrote: post #77
dilyevsky wrote: post #57

Earlier quoted context omitted.

Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code. Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead

I think it happens all the time. There just aren't that many services, relatively speaking, that you'd hear about when this happens. The more-popular services (that you'd hear about) probably already have a bias toward having their stuff better in order, since it's often also really just a matter of making the right investments.

> really just a matter of making the right investments

Right. But the person deciding on those investments is a product owner who's looking to release new features because that's what management wants. At least that's my experience more often than not.

My guess is that the bias might be the opposite. Maybe these companies become more popular because they are reliable and have the mindset to focus on that.

Post reply on HN