Earlier quoted context omitted.
Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.
But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.
An Update on Our Outage
191–200 of 235 posts
Re: An Update on Our Outage
#192Earlier quoted context omitted.
Thank you for posting the text of the blog post. However, I don't understand your TLDR. They clearly were able to identify the root cause, and that is what allowed them to restore service.
People use "root cause" to mean the true underlying driver. You can often restore service without knowing the true root cause. In this example, let's pretend true root cause is a memory leak that takes X days to crash Consul. It's possible they don't know that yet, and just shut down, cleaned up logs and temporary storage, added some capacity, and started back up in a region-by-region ramp up.
Re: An Update on Our Outage
#193Earlier quoted context omitted.
This dynamic service terminology sounds very familiar for someone with a few years in the feature film VFX/Animation industries - however, it would not surprise me if Consol/Vault/Nomad were created without a review of similar situations and solutions in other industries. For those unaware, the feature film VFX/Animation industry has been global and performing large scale technologically ambitious projects requiring…
My mind is continually blown when I read about the sheer volume of innovation from the animation industry over the last 3 decades. I would love to learn more about these scalable solutions you're talking about though, since I'm not familiar with them. Do you have any pointers?
For example, a film compositor I know that is now dead and was revolutionary, called Shake: it pioneered both off loading heavy compute tasks to the GPU (not just graphics, but what is called scientific computing now) and it hid the GCC Compiler inside itself and used a macro-transformed version of C as it's "scripting language" that was actually hot-loaded C++ dlls compiled on the fly. It was also the first "node based programming environment" with graphical nodes the end-users connect with splines to define the I/O between the nodes.
If you are seriously interested, find someone working in the industry today and ask them. That industry has been changing a lot. Since I left the VFX/Animation fields have been moving towards more framework-like production environments, similar to how the web uses frameworks. The issue with these frameworks is they define the tasks to be performed, and those tasks are simplistic and rigid - meaning the actual production work has eliminated in as many places as possible the requirement for an art degree. The VFX/Animation industries are driving production towards something approaching more and more the work of being a burger flipper. The processes are being standardized and reduced in complexity so the studios can hire non-artists, they can hire anyone and work them like an automobile assembly line. Twenty years ago this was starting, and that is when I left.
Re: An Update on Our Outage
#194My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.
This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.
Re: An Update on Our Outage
#195Earlier quoted context omitted.
And that is after you've been working 6 days a week for the last 9 months. As an EA spouse survivor - fuck that industry.
“Spouse survivor”, I like that! I worked on the tech/distribution/data reporting side of the Covid vaccine and consider my wife, and all the sacrifices she made and extra things she handled up on while I was on the keyboard and phone, a critical component of my team’s success.
Re: An Update on Our Outage
#196Earlier quoted context omitted.
So true. And also at the systems level. In most places I've seen, the #1 priority is hitting arbitrary executive feature/date goals, not maintaining robust systems. At some point, the shit will hit the fan, causing a "Why didn't you do perfectly the thing that wasn't a real priority?!?" reaction and a temporary lurch toward robustness. Although often the lurch will be less about actual robustness and more toward perf…
I worked at a big fintech company about to go public in a SPAC worth $4 billion. They were a unicorn when I was there. Literally everything was in a GCP MySQL Database......and they refused to pay for a hot spare. They had backups but nothing for redundancy. We had downtime almost every week.
Re: An Update on Our Outage
#197Earlier quoted context omitted.
Are you implying Netflix is not a tech company?
I definitely see Netflix as a Media Company, much like Disney. Disney and Pixar have great tech too, but they are not tech company in any shape or form. How Netflix got lumped into "FAANG" aka Big Tech is still a mystery to me.
In some sense, I agree - Netflix, Disney, and Pixar are media companies that use technology to distribute, to recommend, and to produce their content.
But in the same sense, Google is an advertising company that uses technology to sell and to show their ads (and they have a few loss-leading tech ventures like search, mail, a phone OS, and a browser - all of which they give away to more effectively show those ads). Facebook is another advertising company, they happen to show and profile their ads using their social network tech. Amazon is a retail sales, warehousing, and product distribution company that happens to use a lot of technology. They also have a "tech company" division that sells web services, but that's only 13% of their revenue.
Similarly, Walmart is a retail sales and warehousing company (as well as product distribution now, but most recognizable by their brick-and-mortar stores) that uses technology in every aspect of buying product, shipping it to stores, deciding where and how much of it to stock, how to price it, and when to restock it, just like Amazon. There was a time when Sam Walton walked around, squinted at the shelves, and made notes about what and how much to order on a clipboard, but that's long past - it's all tech now. Automotive manufacturers are other very big tech companies, they build and sell automobiles but every step of that design, fabricate, market, and sell process uses technology.
Apple, at least, sells technology hardware. Microsoft also sells technology hardware and software, but they did so before FAANG were big so they're not in the list either.
The list is arbitrary. Software and technology is eating - has eaten - the world, and management of it is critical to every business.
Re: An Update on Our Outage
#198Earlier quoted context omitted.
That behavior is a reflection of their citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.
>”citizenry who have been convinced by propaganda that government can never be effective at anything and therefore should not be funded properly.” People are treating the Texas ice storm like it was some comeuppance event but the truth of the matter is that Texas rarely experiences this kind of winter weather and that’s the reason why it didn’t handle the big freeze the same way as a midwestern or northeastern state…
In the 2021 storm, hundreds of people died. Comeuppance is not the right word. I don't know the right one, but I believe it's closer to willful blindness or predatory delay.
[0] https://www.ferc.gov/sites/default/files/2020-04/08-16-11-re...
Re: An Update on Our Outage
#199Earlier quoted context omitted.
But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.
There's nothing intrinsically wrong with say ZK, it's very powerful, but my point is that you should avoid these systems if you can. Distributed concensus is not an easy problem, it's very error prone, and if you do this more than you absolutely have to you're doing it wrong.
Re: An Update on Our Outage
#200My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…
Consul is insanely easy to fix if you keep backups. It is as simple as deploying a brand new cluster and restoring a known good snapshot.
Sadly, a surprising amount of practitioners don't keep backups.