Live data from Hacker News

An Update on Our Outage

blog.roblox.com

161–170 of 235 posts

Re: An Update on Our Outage

#161

Earlier quoted context omitted.

Not a software outage, but the Texas ice storm of February resulted in me not receiving mail for two weeks, not being able to use the streets for at least a week (they never plowed and it just eventually melted), rolling blackout where I had power for about four hours a day, and no Internet service at all for the first two days. No water for most of the first day. I'd argue keeping basic city services and roads opera…

That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.

>”citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.”

People are treating the Texas ice storm like it was some comeuppance event but the truth of the matter is that Texas rarely experiences this kind of winter weather and that’s the reason why it didn’t handle the big freeze the same way as a midwestern or northeastern state would.

The reason why cities like DFW do not maintain a fleet of snowplows isn't because of anti-government propaganda, but because it is completely uneconomical to do so. Even during the big freeze there wasn’t a foot of snow on the ground. The major roadways were salted, though, and you could drive on them.

Snow and icing events tend to only happen for a day or two each year in my part of Texas. And it is often the case that when temperatures do dip below zero there isn’t preceding rain or snow. But when that does happen, most people just ride it out for a day or two and wait for things to melt.

Re: An Update on Our Outage

#162
post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

We have hospitals being knocked out for weeks after ransomware attacks. If we complain about roblox I think we should reflect on our priorities.

Why not complain about both?

Re: An Update on Our Outage

#163
post #159

Earlier quoted context omitted.

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

By that logic Facebook certainly wouldn't be a tech company either.

Re: An Update on Our Outage

#164
post #159

Earlier quoted context omitted.

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

Now probably, but at the time the FAANG acrnonym was conceived, Netflix Originals and other productions ran internally was a drop in the ocean compared to their licensed content, similar to how Apple has controlled music (before streaming), so it made sense.

Re: An Update on Our Outage

#165

Earlier quoted context omitted.

We have hospitals being knocked out for weeks after ransomware attacks. If we complain about roblox I think we should reflect on our priorities.

Why not complain about both?

You can, wasn't the point I tried to make. I tried to point out the arbitrary expectations here. Depending on the fault such downtime can last a few hours to a few weeks. I don't think you can make conclusions on architecture in this case.

Perhaps it would be possible to roll back everything in just a few minutes but some users might lose a lot of work. If such special cases are to be resolved, it may take a lot of time. You just cannot really infer quality of architecture by downtime alone.

Re: An Update on Our Outage

#166
post #125
post #121

Earlier quoted context omitted.

I don't know, anecdotally it seems to me that Consul is more popular than Zookeeper. But yeah, you should go with the one you know best, especially at any sort of scale. Vault is the most popular secrets manager out there, and has a lot of advantages over something like Kubernetes secrets ( which aren't even encrypted). Nomad is a bit obscure but IMHO it seems to be gaining momentum.

ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint. Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki. When it comes to which one has the most production hours und…

Lot of people moved away from ZK because it used to be a nightmare to setup and maintain, ZK is not vastly more popular than Consul.

In every places I worked in the last 5y we did not even considered ZK because of the setup and all the components it requires.

Re: An Update on Our Outage

#167
post #159

Earlier quoted context omitted.

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

Because FAANG was coined as a term for super high growth stocks, not for anything to do with technology.

Re: An Update on Our Outage

#168

Earlier quoted context omitted.

Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…

This dynamic service terminology sounds very familiar for someone with a few years in the feature film VFX/Animation industries - however, it would not surprise me if Consol/Vault/Nomad were created without a review of similar situations and solutions in other industries. For those unaware, the feature film VFX/Animation industry has been global and performing large scale technologically ambitious projects requiring…

My mind is continually blown when I read about the sheer volume of innovation from the animation industry over the last 3 decades. I would love to learn more about these scalable solutions you're talking about though, since I'm not familiar with them. Do you have any pointers?

Re: An Update on Our Outage

#169
> we will implement a policy to make our creator community economically whole as a result of this outage.

Mad props for this. Who else would do such a thing for an outage? I wonder how they will do this of course. The devil will be in the details :)

Re: An Update on Our Outage

#170
post #163
post #159

Earlier quoted context omitted.

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

By that logic Facebook certainly wouldn't be a tech company either.

Why not? Didn't Facebook create tools like GraphQL and React?
Post reply on HN