Live data from Hacker News

An Update on Our Outage

blog.roblox.com

191–200 of 235 posts

Re: An Update on Our Outage

#191

Earlier quoted context omitted.

Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.

But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.

There's nothing intrinsically wrong with say ZK, it's very powerful, but my point is that you should avoid these systems if you can. Distributed concensus is not an easy problem, it's very error prone, and if you do this more than you absolutely have to you're doing it wrong.

Re: An Update on Our Outage

#192
post #173

Earlier quoted context omitted.

Thank you for posting the text of the blog post. However, I don't understand your TLDR. They clearly were able to identify the root cause, and that is what allowed them to restore service.

People use "root cause" to mean the true underlying driver. You can often restore service without knowing the true root cause. In this example, let's pretend true root cause is a memory leak that takes X days to crash Consul. It's possible they don't know that yet, and just shut down, cleaned up logs and temporary storage, added some capacity, and started back up in a region-by-region ramp up.

Fine, but from their blog post they pretty well did describe the root cause. I don't expect them to post code line fixes in a public blog post.

Re: An Update on Our Outage

#193

Earlier quoted context omitted.

This dynamic service terminology sounds very familiar for someone with a few years in the feature film VFX/Animation industries - however, it would not surprise me if Consol/Vault/Nomad were created without a review of similar situations and solutions in other industries. For those unaware, the feature film VFX/Animation industry has been global and performing large scale technologically ambitious projects requiring…

My mind is continually blown when I read about the sheer volume of innovation from the animation industry over the last 3 decades. I would love to learn more about these scalable solutions you're talking about though, since I'm not familiar with them. Do you have any pointers?

I've not worked in VFX since '02, but I was both at Rhythm & Hues Studios through multiple VFX Oscars as well as an early 3D graphics researcher during the 80's. I have no idea which tools are still around.

For example, a film compositor I know that is now dead and was revolutionary, called Shake: it pioneered both off loading heavy compute tasks to the GPU (not just graphics, but what is called scientific computing now) and it hid the GCC Compiler inside itself and used a macro-transformed version of C as it's "scripting language" that was actually hot-loaded C++ dlls compiled on the fly. It was also the first "node based programming environment" with graphical nodes the end-users connect with splines to define the I/O between the nodes.

If you are seriously interested, find someone working in the industry today and ask them. That industry has been changing a lot. Since I left the VFX/Animation fields have been moving towards more framework-like production environments, similar to how the web uses frameworks. The issue with these frameworks is they define the tasks to be performed, and those tasks are simplistic and rigid - meaning the actual production work has eliminated in as many places as possible the requirement for an art degree. The VFX/Animation industries are driving production towards something approaching more and more the work of being a burger flipper. The processes are being standardized and reduced in complexity so the studios can hire non-artists, they can hire anyone and work them like an automobile assembly line. Twenty years ago this was starting, and that is when I left.

Re: An Update on Our Outage

#194
post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

From my experience there are a few engineers that are usually key to fixing such issues (not always the same people, depending on the failure). Handing-off work is more for day to day work like handeling a support ticket. So the key engineers need to work crazy hours when things like that happen. But that's OK, ideally these things are rare and people need to work crazy hours only in those rare situations. I'm super protective about my evening/weekend time but I wouldn't have any problem working the entire weekend like crazy when something like this happens. If it happens too often I will consider finding a better place to work.

Re: An Update on Our Outage

#195

Earlier quoted context omitted.

And that is after you've been working 6 days a week for the last 9 months. As an EA spouse survivor - fuck that industry.

“Spouse survivor”, I like that! I worked on the tech/distribution/data reporting side of the Covid vaccine and consider my wife, and all the sacrifices she made and extra things she handled up on while I was on the keyboard and phone, a critical component of my team’s success.

"EA Spouse" is the name is a specific period during the early 90's when Electronic Arts burned out an entire generation of game developers.

Re: An Update on Our Outage

#196

Earlier quoted context omitted.

So true. And also at the systems level. In most places I've seen, the #1 priority is hitting arbitrary executive feature/date goals, not maintaining robust systems. At some point, the shit will hit the fan, causing a "Why didn't you do perfectly the thing that wasn't a real priority?!?" reaction and a temporary lurch toward robustness. Although often the lurch will be less about actual robustness and more toward perf…

I worked at a big fintech company about to go public in a SPAC worth $4 billion. They were a unicorn when I was there. Literally everything was in a GCP MySQL Database......and they refused to pay for a hot spare. They had backups but nothing for redundancy. We had downtime almost every week.

Guessing its SoFi.

Re: An Update on Our Outage

#197
post #159

Earlier quoted context omitted.

Are you implying Netflix is not a tech company?

I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.

It's hard to define any company as a 'tech company' if you're willing to define Netflix as a Media Company.

In some sense, I agree - Netflix, Disney, and Pixar are media companies that use technology to distribute, to recommend, and to produce their content.

But in the same sense, Google is an advertising company that uses technology to sell and to show their ads (and they have a few loss-leading tech ventures like search, mail, a phone OS, and a browser - all of which they give away to more effectively show those ads). Facebook is another advertising company, they happen to show and profile their ads using their social network tech. Amazon is a retail sales, warehousing, and product distribution company that happens to use a lot of technology. They also have a "tech company" division that sells web services, but that's only 13% of their revenue.

Similarly, Walmart is a retail sales and warehousing company (as well as product distribution now, but most recognizable by their brick-and-mortar stores) that uses technology in every aspect of buying product, shipping it to stores, deciding where and how much of it to stock, how to price it, and when to restock it, just like Amazon. There was a time when Sam Walton walked around, squinted at the shelves, and made notes about what and how much to order on a clipboard, but that's long past - it's all tech now. Automotive manufacturers are other very big tech companies, they build and sell automobiles but every step of that design, fabricate, market, and sell process uses technology.

Apple, at least, sells technology hardware. Microsoft also sells technology hardware and software, but they did so before FAANG were big so they're not in the list either.

The list is arbitrary. Software and technology is eating - has eaten - the world, and management of it is critical to every business.

Re: An Update on Our Outage

#198

Earlier quoted context omitted.

That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.

>”citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.” People are treating the Texas ice storm like it was some comeuppance event but the truth of the matter is that Texas rarely experiences this kind of winter weather and that’s the reason why it didn’t handle the big freeze the same way as a midwestern or northeastern state…

NERC created a report after the Texas 2011 ice storm power outage with recommendations for ERCOT generators to improve their weatherization standards [0]. In that report they referenced not only the 2011 incident but previous incidents where the grid generators failed to adequately prepare for potential problems due to weather related issues.

In the 2021 storm, hundreds of people died. Comeuppance is not the right word. I don't know the right one, but I believe it's closer to willful blindness or predatory delay.

[0] https://www.ferc.gov/sites/default/files/2020-04/08-16-11-re...

Re: An Update on Our Outage

#199

Earlier quoted context omitted.

But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.

There's nothing intrinsically wrong with say ZK, it's very powerful, but my point is that you should avoid these systems if you can. Distributed concensus is not an easy problem, it's very error prone, and if you do this more than you absolutely have to you're doing it wrong.

You are 100% correct. Avoid at all costs but when you need it, well you need it. Usually if you find yourself in that boat the best thing you can do is just understand exactly why you need consensus and ensure scope of data managed in ZK never expands beyond that point.

Re: An Update on Our Outage

#200
post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

> For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against.

Consul is insanely easy to fix if you keep backups. It is as simple as deploying a brand new cluster and restoring a known good snapshot.

Sadly, a surprising amount of practitioners don't keep backups.

Post reply on HN