Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

71–80 of 235 posts

Re: An Update on Our Outage

#71

Post by slg »

kaesar14 wrote: post #47
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

The last outage of this kind I can even remember is the summer PSN was down for like, a month?

Although that PSN downtime probably fails into a different category as it was a response to being hacked. Sony intentionally took the service down and kept it down to investigate and fix their security holes. Basically the equivalent of temporarily closing a brick and mortar store after a robbery compared to being involuntarily closed because no one could find the right set of keys to unlock the store's front door.

Re: An Update on Our Outage

#72

Post by zz865 »

> performance tuning, re-configuration, and scaling back of some load.

That seems to be a great thing if you can sit at the edge and scale the number of connections coming in. I just wish our system could do that.

Re: An Update on Our Outage

#73

Post by knownjorbist »

vmception wrote: post #23
danso wrote: post #16

It’s great to hear that it wasn’t the result of something malicious (e.g. a hack). But this has to be one of the longest outages by a company this big ($50B market cap), at least in the past decade it seems?

What I love most then is that its the largest and it doesn't matter! In this market they will become an even larger company because competency has no correlation to how much money subscribers/advertisers will pay, and none of that has any correlation to what investors will pay! Anybody optimizing for competence is exchanging time for food and shelter and just gets to be reminded of the much harder game they are playi…

Let's reserve this level of vitriol for medical systems or powerplants, shall we?

Re: An Update on Our Outage

#75

Post by rStar »

kaesar14 wrote: post #47
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

The last outage of this kind I can even remember is the summer PSN was down for like, a month?

looks familiar to me also. was roblox also hacked?

Re: An Update on Our Outage

#77

Post by sverhagen »

dilyevsky wrote: post #57
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code. Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead

I think it happens all the time. There just aren't that many services, relatively speaking, that you'd hear about when this happens. The more-popular services (that you'd hear about) probably already have a bias toward having their stuff better in order, since it's often also really just a matter of making the right investments.

Re: An Update on Our Outage

#78

Post by shoghicp »

mr-karan wrote: post #4

Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…

You can read the blog post via the... RSS feed!

https://blog.roblox.com/feed/

Re: An Update on Our Outage

#79

Post by mmmBacon »

_nickwhite wrote: post #9

My 3 youngest kids, aged 9, 9, and 8 told me Roblix were having an extended outage, world wide. Good job to the team getting everything back up. Been there, done that. #hugops

My 8 y.o. daughter ran through the house with an excitement typically I only see on Christmas morning exclaiming “Roblox is back!”

Re: An Update on Our Outage

#80

Post by sam0x17 »

busymom0 wrote: post #69
sam0x17 wrote: post #65

> A key value at Roblox is “Respect the Community,” Yet they still take a 30% cut on all transactions that go to content creators / developers

That high percentage doesn’t mean they do or do not respect the community. Saying this as someone who doesn’t know much about Roblox but does give 15-30% to Apple and Google. Google definitely doesn’t respect their developers. Apple kind of does.

Apple and Google also disrespect their developer communities, probably more than any companies out there. Anything over 5% is ridiculous and was seen as such originally before Google and Apple made this the norm.
Post reply on HN