141–150 of 184 posts
This was a nice short read. A simple (temporary) solution, yet a clever one. How was he managing the instances? Was he using kubernetes, or did he write some script to manage the auto terminating of the instances? It would also be nice to know why: 1. Killing was quicker than restarting. Perhaps because of the business logic built into the java application? 2. Killing was safe. How was the system architectured so tha…
These companies have achieved vast scale because correctness doesn’t matter that much so long as it is “good enough” for a large enough statistical population, and their Devops practices and coding practices have evolved with this as a key factor.
It is not uncommon at all for Netflix or Hulu or Facebook or Instagram to throw an error or do something bone headed. When it happens you shrug and try again.
Now imagine if this was applied to credit card payments systems, or your ATM network, or similar. The reality of course is that some financial systems do operate this way, but it’s recognized as a problem and usually gets on people’s radar to fix as failed transaction rates creep up and it starts costing money directly or clients.
“Just randomly kill shit” is perfectly fine in the Netflix world. In other domains, not so much (but again it can and will be used as an emergency measure!).
Earlier quoted context omitted.
In cases like this won't Chaos Monkey actually hide the problem, since it's basically doing exactly the same as their mitigation strategy - randomly restarting services?
Right. The point of the question is why not ramp up the monkey? They seem to imply it isn’t there now, which wouldn’t surprise me with the cultural shifts that have occurred in the tech world.
I understand how their approach worked well enough, but I don’t get why they couldn’t selectively target the VMs that were currently experiencing problems rather than randomly select any VM to terminate. If they were exhausting all their CPU resources, wouldn’t that be easy enough to search for using something like ansible?
My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…
Sounds like process-level garbage collection. Just kill it and restart. Which also sound like the apocryphal tale about the leaky code and the missile. "This sparked and interesting memory for me. I was once working with a customer who was producing on-board software for a missile. In my analysis of the code, I pointed out that they had a number of problems with storage leaks. Imagine my surprise when the customers c…
Vaguely related anecdote: 30 years ago or so I worked at a tiny networking company where several coworkers came from a small company (call it C) that made AppleTalk routers. They recounted being puzzled that their competitor (company S) had a reputation for having a rock-solid product, but when they got it into the lab they found their competitor's product crashed maybe 10 times more often than their own. It turned o…
Earlier quoted context omitted.
We are talking about Netflix. You know, the 'N' in FAANG/MAANG or whatever.
As a non-FAANGer Netflix has always intrigued me because of this. While Google, Facebook and others seem to have bogged themselves down in administrative mess, Netflix still seems agile. From the outside at least. (also worth noting this post seems to be discussing an event that occurred many years ago, circa 2011, so might not be a reflection of where they are today)
Earlier quoted context omitted.
We are talking about Netflix. You know, the 'N' in FAANG/MAANG or whatever.
As a non-FAANGer Netflix has always intrigued me because of this. While Google, Facebook and others seem to have bogged themselves down in administrative mess, Netflix still seems agile. From the outside at least. (also worth noting this post seems to be discussing an event that occurred many years ago, circa 2011, so might not be a reflection of where they are today)
> It was Friday afternoon > Rolling back was cumbersome It's a fundamental principle of modern DevOps practice that rollbacks should be quick and easy, done immediately when you notice a production regression, and ideally automated. And at Netflix's scale, one would have wanted this rollout to be done in waves to minimize risk. Apparently this happened back in 2021. Did the team investigate later why you couldn't do…
Then DevOps principles are in conflict with reality.