Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

111–120 of 235 posts

Re: An Update on Our Outage

#111

Post by xxs »

colordrops wrote: post #84
inside_edition wrote: post #55

Earlier quoted context omitted.

Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…

> Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?

the thing can even provide dns... but it has fancy interfaces and an http endpoint. Technically you could do everything with dns just fine.

Even if consul provides some sort of liveness (due to latency and concurrency it means little), I can't think of a usable case where the clients won't have retries and ability to maintain multiple open sockets, etc.

Re: An Update on Our Outage

#112

Post by jpgvm »

strictfp wrote: post #107
jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

My stance is: why even have any of those? Are you running in a local datacenter?

Generally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism.

If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB w/conditional writes, k8s configmap).

However for what these systems do best, i.e very high performance distributed locking and consistent metadata with fast reads w/watches and relative fast writes there is no replacement. You either need one of ZK/etcd/consul/something else based on raft/paxos/zab.

EDIT: I realize now you might be talking about Nomad/Vault also.

So in this case I think they are running their own DCs on bare metal and they opted for Nomad over k8s. Vault is somewhat special because it has capabilities no other secrets system does out of the box, specifically it has integrations to create short-lived credentials on the fly for clients - so called dynamic secrets have many advantages and is why services like AWS STS are so popular (the magic sauce behind AssumeRole).

Re: An Update on Our Outage

#113

Post by jpgvm »

faizshah wrote: post #103
jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

Just curious, what are the common ZK failure modes?

In high scale situations, somewhat common if you aren't ontop of metrics etc.

In most other cases it's just people not turning on auto-purge and having disks fill up with snapshots.

So it varies depending on your teams experience with ZK and/or if you are using some sort of community managed deployment primitive that has sensible defaults baked in.

Re: An Update on Our Outage

#114

Post by withinboredom »

weird-eye-issue wrote: post #85
sam0x17 wrote: post #81

Earlier quoted context omitted.

Anything over 5% is highway robbery and the kind of thing that only exists in concerningly unregulated capitalism. I promise you someday legislation will put an actual legal limit there. Just wait for these zoomers to reach voting age.

Payment processors charge around 3% plus fees (charge back fees and transaction fees). I think the value that Roblox adds makes them deserve much more than a measly 2% more than a generic payment processor would get. 15 or 20% is probably a good middle ground

I assume they have a deal with Apple and Google for less than 30% on in-app transactions. That probably means Roblox is getting between 10-20% of the transaction.

Re: An Update on Our Outage

#115

Post by frabcus »

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

My favourite was the Skype outage in 2007 where a peer to peer algorithm didn’t scale - they had to work this out, and change the algorithm to fix it. It took ages!

https://www.wired.com/2007/08/update-skype-ou/

Re: An Update on Our Outage

#116

Post by nodejs_rulez_1 »

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

I am sure they got a lot of stock.

Re: An Update on Our Outage

#117

Post by Cthulhu_ »

nodejs_rulez_1 wrote: post #116
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

I am sure they got a lot of stock.

I mean I'd gladly pull an all-nighter if I were a millionaire thanks to the thing I'm fixing. I think. Then I'd quit and do something leisurely.

Re: An Update on Our Outage

#118

Post by weird-eye-issue »

withinboredom wrote: post #114
weird-eye-issue wrote: post #85

Earlier quoted context omitted.

Payment processors charge around 3% plus fees (charge back fees and transaction fees). I think the value that Roblox adds makes them deserve much more than a measly 2% more than a generic payment processor would get. 15 or 20% is probably a good middle ground

I assume they have a deal with Apple and Google for less than 30% on in-app transactions. That probably means Roblox is getting between 10-20% of the transaction.

They also have PC and Mac version of it. I have some friends who have played Roblox on PC and my younger cousin plays on iOS. I wonder what the market share looks like between them

Re: An Update on Our Outage

#119

Post by swader999 »

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

Not of this size or scale but ransomeware attacks typically take days to resolve. The pipeline attack and the large multinational vet company come to mind.

Re: An Update on Our Outage

#120

Post by swader999 »

knownjorbist wrote: post #73
vmception wrote: post #23

Earlier quoted context omitted.

What I love most then is that its the largest and it doesn't matter! In this market they will become an even larger company because competency has no correlation to how much money subscribers/advertisers will pay, and none of that has any correlation to what investors will pay! Anybody optimizing for competence is exchanging time for food and shelter and just gets to be reminded of the much harder game they are playi…

Let's reserve this level of vitriol for medical systems or powerplants, shall we?

Some level of intensity is warranted. Things got real pretty fast here this past week when my eight year old was home on a school PD day.
Post reply on HN