Live data from Hacker News

An Update on Our Outage

blog.roblox.com

221–230 of 235 posts

Re: An Update on Our Outage

#221
post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

> What was the last service of this size that was down for 4 days

Was "only" two days, but Fortnite just shutdown for 2 days as a part of an end-of-season event back in 2019 https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit...

https://www.polygon.com/fortnite/2019/10/13/20911691/fortnit...

Re: An Update on Our Outage

#222
post #159

Earlier quoted context omitted.

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

> How Netflix got lumped into

Otherwise it would be a slur.

Re: An Update on Our Outage

#223
post #159

Earlier quoted context omitted.

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

Two ways I would argue for Netflix being a tech company.

A critical question is, how and how much do they pay their developers (and other tech folks)? I believe Netflix compensates their tech workers more richly than the average media company. I am certain of it if you take an average over the past 10 years.

The other argument is, when we say tech, we really mean qualitative innovation. Did Netflix disrupt an existing 800 pound gorilla using technology? The answer is yes, they killed Blockbuster by embracing tech.

Re: An Update on Our Outage

#224
post #130
post #125

Earlier quoted context omitted.

ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint. Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki. When it comes to which one has the most production hours und…

ZK isn't sexy, we agree. IIRC Kafka no longer requires it? And IIRC it was one of the main downsides of Kafka, that it required a whole ZK cluster. etcd doesn't compare directly to ZK and Consul, because it's just a distributed KV store. You can build service discovery on top of it with custom tooling, but doesn't come out of the box. Consul does that + service discovery + service mesh + (since last week ) API gatewa…

Kafka still requires it, at least for all production clusters. In a few years the recommendation might be the new internal Raft based consensus though.

Re: An Update on Our Outage

#225
post #87

Earlier quoted context omitted.

Why is making 6% from your one funding source on an otherwise free-to-play product unreasonable? Do you think any markup of more than 5% is unreasonable? What about Apple's physical products? What about ... every retail store in existence? > Just wait for these zoomers to reach voting age. Ah yes, 50.

> Do you think any markup of more than 5% is unreasonable? What about Apple's physical products? What about ... every retail store in existence? Retail stores maintain a physical presence and have actual overhead, as do online retailers that have to maintain warehouses, subsidize free shipping, etc. What is the overhead on an app submission, or an app purchase? Charging 30% overhead for $0.0012 cents of bandwidth, an…

> In the case of Roblox, you can argue "but what about the platform?" but 5% is still a metric-crap-ton of revenue and is more than enough to keep their platform afloat (especially if they don't have to give 30% upstream to Apple/Google)

Afloat? Maybe. Seems doubtful. Anyway, the justification for app stores charging developers is always providing the Platform. Android is a free platform. You can buy an Android phone from a manufacturer of your choice without Google seeing a cent of that money. Their way of making money is to use rents on their application store.

What I'm getting at is that they produce a product. A valuable product, given that many people enjoy it. Under standard American assumptions about free markets, this means they're entitled to a reasonable profit, not just enough to "keep their platform afloat". Profit is not theft.

Think about a company that sells you a printer at-cost in the hope that it will pay for itself in profits when you buy ink. There's usually third-party ink that will work in the printer, but the business model of the company is that there will be enough people who buy their ink to make cheap printers profitable. There's a, I don't know, 150% markup on the ink.

In other words, they have an overall profitable business, where one part of it is a cost leader and another is where they extract rents. Nothing particularly surprising about this. The app store model is exactly the same: most parts of the platform are free (despite very real development costs), and store developers pay a fee of 20-30% which keeps things running. In fact, if the developers reduced the fee but then charged users to use the platform, the overall profits of developers would probably be reduced, as fewer users would use the platform.

I'm skeptical, in the end, that the profits earned by companies like Roblox are terribly out of keeping with those owned by retail stores or computer manufacturers. You cannot separate out the parts of the business that are pure profit centers from those that are cost leaders. The fact that the profit on the profit centers seems exorbitant has to be taken in context.

Re: An Update on Our Outage

#226

Earlier quoted context omitted.

> My 3 youngest kids, aged 9, 9, and 8 ... That must have been quite a year in the White household!

Twins, and then probably ~18 months later a younger child.

Twins + adoption actually. The current picture is 8, 9, 9, 16, 16. Figure that one out. :)

Re: An Update on Our Outage

#227

Earlier quoted context omitted.

So true. And also at the systems level. In most places I've seen, the #1 priority is hitting arbitrary executive feature/date goals, not maintaining robust systems. At some point, the shit will hit the fan, causing a "Why didn't you do perfectly the thing that wasn't a real priority?!?" reaction and a temporary lurch toward robustness. Although often the lurch will be less about actual robustness and more toward perf…

I worked at a big fintech company about to go public in a SPAC worth $4 billion. They were a unicorn when I was there. Literally everything was in a GCP MySQL Database......and they refused to pay for a hot spare. They had backups but nothing for redundancy. We had downtime almost every week.

[deleted]

Re: An Update on Our Outage

#228

Earlier quoted context omitted.

That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.

>”citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.” People are treating the Texas ice storm like it was some comeuppance event but the truth of the matter is that Texas rarely experiences this kind of winter weather and that’s the reason why it didn’t handle the big freeze the same way as a midwestern or northeastern state…

So. Do you think Texas should winterize or not?

Re: An Update on Our Outage

#229

Earlier quoted context omitted.

I wonder how long it will take until people adopt their new name, and MAANG

might as well go for MAMAG at that point, swapping Netflix out and Microsoft in. Or MAMANG. Or even AMMANG. Possibilities are endless. :)

MAGMA. You heard it here first

Re: An Update on Our Outage

#230

Earlier quoted context omitted.

I don't disagree with you. I was just stating that Google is absolutely horrible whereas Apple at least let's us developers talk to a human being.

Apple is the worst of them all. They are much more restrictive and arbitrary with their app store policies (see previous HN articles over the last 5 years), they have done much more to lock down their devices to prevent circumvention of things like banning emulators, and they don't license the apple device model to other vendors, so they are the true definition of a walled garden. Google has plenty of problems too, b…

You aren't wrong.

Though I was mostly talking about customer service part and development relations in terms of that.

Google regularly bans developers for absurd reasons or no reasons at all, doesn't let them talk to any humans until the developer causes enough twitter storm and gets some tech sites to write about it to get some attention from Google. Apple at least lets the dev talk to human beings.

Post reply on HN