Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

211–220 of 235 posts

Re: An Update on Our Outage

#211

Post by sam0x17 »

weird-eye-issue wrote: post #85
sam0x17 wrote: post #81

Earlier quoted context omitted.

Anything over 5% is highway robbery and the kind of thing that only exists in concerningly unregulated capitalism. I promise you someday legislation will put an actual legal limit there. Just wait for these zoomers to reach voting age.

Payment processors charge around 3% plus fees (charge back fees and transaction fees). I think the value that Roblox adds makes them deserve much more than a measly 2% more than a generic payment processor would get. 15 or 20% is probably a good middle ground

Right, that's why they get an extra 2%.

Re: An Update on Our Outage

#212

Post by sam0x17 »

busymom0 wrote: post #83
sam0x17 wrote: post #80

Earlier quoted context omitted.

Apple and Google also disrespect their developer communities, probably more than any companies out there. Anything over 5% is ridiculous and was seen as such originally before Google and Apple made this the norm.

I don't disagree with you. I was just stating that Google is absolutely horrible whereas Apple at least let's us developers talk to a human being.

Apple is the worst of them all. They are much more restrictive and arbitrary with their app store policies (see previous HN articles over the last 5 years), they have done much more to lock down their devices to prevent circumvention of things like banning emulators, and they don't license the apple device model to other vendors, so they are the true definition of a walled garden. Google has plenty of problems too, but at least random device companies can make android devices, and third party app stores are (sort of) allowed to exist.

Re: An Update on Our Outage

#213

Post by tyingq »

hn_throwaway_99 wrote: post #192
tyingq wrote: post #173

Earlier quoted context omitted.

People use "root cause" to mean the true underlying driver. You can often restore service without knowing the true root cause. In this example, let's pretend true root cause is a memory leak that takes X days to crash Consul. It's possible they don't know that yet, and just shut down, cleaned up logs and temporary storage, added some capacity, and started back up in a region-by-region ramp up.

Fine, but from their blog post they pretty well did describe the root cause. I don't expect them to post code line fixes in a public blog post.

Perhaps, though "prompted by a subtle bug in our backend service communications while under heavy load" is pretty opaque.

Re: An Update on Our Outage

#214

Post by sam0x17 »

bscphil wrote: post #87
sam0x17 wrote: post #81

Earlier quoted context omitted.

Anything over 5% is highway robbery and the kind of thing that only exists in concerningly unregulated capitalism. I promise you someday legislation will put an actual legal limit there. Just wait for these zoomers to reach voting age.

Why is making 6% from your one funding source on an otherwise free-to-play product unreasonable? Do you think any markup of more than 5% is unreasonable? What about Apple's physical products? What about ... every retail store in existence? > Just wait for these zoomers to reach voting age. Ah yes, 50.

> Do you think any markup of more than 5% is unreasonable? What about Apple's physical products? What about ... every retail store in existence?

Retail stores maintain a physical presence and have actual overhead, as do online retailers that have to maintain warehouses, subsidize free shipping, etc. What is the overhead on an app submission, or an app purchase? Charging 30% overhead for $0.0012 cents of bandwidth, an algorithmic review of your app that only might be supplemented by an actual human review, and a literally free file copy operation comes nowhere close to that. This is a lot closer to payment processor fee territory in terms of the actual work being done by the provider.

You would think they would do more for their app developers with such a hefty fee, but they don't, so they need to be regulated. These fees are just theft.

In the case of Roblox, you can argue "but what about the platform?" but 5% is still a metric-crap-ton of revenue and is more than enough to keep their platform afloat (especially if they don't have to give 30% upstream to Apple/Google). We've just become accustomed to ridiculous % cuts because monopolies have normalized this practice. In the early 00s, the idea of an app store even taking a cut was viewed as laughable. Then it became normal.

Re: An Update on Our Outage

#215

Post by xorcist »

op00to wrote: post #139
xorcist wrote: post #128

Earlier quoted context omitted.

> > We didn’t want to choose any technology that requires the company to drive deep expertise, That's a beautifully concise quote which neatly summarizes what contemporary IT values.

You don’t need deep etcd experience to run k8s. Why is that bad?

You don't need more than high school science just to build a bridge. Yet we keep educating engineers. Completely unnecessary, as long as nothing breaks.

You don't really need to understand the citric acid cycle just to operate on someone. Yet we keep educating surgeons. Is that a bad thing?

Re: An Update on Our Outage

#216

Post by kerng »

867-5309 wrote: post #204
kerng wrote: post #163

Earlier quoted context omitted.

By that logic Facebook certainly wouldn't be a tech company either.

I wonder how long it will take until people adopt their new name, and MAANG

The new offical acronym on the street is MAMAA by the way. Microsoft, Apple, Meta, Amazon and Alphabet.

Jim Cramer announced this on Friday.

Re: An Update on Our Outage

#217

Post by munificent »

bsenftner wrote: post #195
chasd00 wrote: post #154

Earlier quoted context omitted.

“Spouse survivor”, I like that! I worked on the tech/distribution/data reporting side of the Covid vaccine and consider my wife, and all the sacrifices she made and extra things she handled up on while I was on the keyboard and phone, a critical component of my team’s success.

"EA Spouse" is the name is a specific period during the early 90's when Electronic Arts burned out an entire generation of game developers.

Here is the source of the term:

https://ea-spouse.livejournal.com/274.html

Re: An Update on Our Outage

#218

Post by xbar »

I am disappointed by the lack of technical transparency in this update. If Roblox knew enough about the problem to address it, they could have included enough technical words to describe it.

Which piece was overwhelmed and in which piece of backend did the subtle bug exist?

A deeper post-mortem will be nice. An actual description of the problem seems like SRE table-stakes for outages.

Re: An Update on Our Outage

#219

Post by presidentender »

miketery wrote: post #196
gsibble wrote: post #185

Earlier quoted context omitted.

I worked at a big fintech company about to go public in a SPAC worth $4 billion. They were a unicorn when I was there. Literally everything was in a GCP MySQL Database......and they refused to pay for a hot spare. They had backups but nothing for redundancy. We had downtime almost every week.

Guessing its SoFi.

SoFi's devops function is extremely robust, if overcomplicated, and they've got redundancy for all their application servers and production databases. They're also an AWS shop, not GCP.

Re: An Update on Our Outage

#220

Post by tofuahdude »

xbar wrote: post #218

I am disappointed by the lack of technical transparency in this update. If Roblox knew enough about the problem to address it, they could have included enough technical words to describe it. Which piece was overwhelmed and in which piece of backend did the subtle bug exist? A deeper post-mortem will be nice. An actual description of the problem seems like SRE table-stakes for outages.

They do say in the post that they're going to explain more, later.
Post reply on HN