Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

111–120 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#111

Post by TZubiri »

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure.

But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like when users stop watching, and how to recommend them titles to keep them hooked, and when to cut the titles and autoplay the next video to make them addicted to binge watching.

Case in point, the blogpost speaks of a CPU concurrency bug and clients being servers? But never once refers to an actual business domain purpose. Like are these servers even loading video content? My bet is they are more on the optimizing engagement side of things. And I make this bet knowing that these are servers with high video-like load, but I'm confident that these guys are juggling 10TB/s of mouse metadata into some ML system more than I'm confident that they have some problem with the core of their technology which has worked since launch.

As I say this, I know I'm probably wrong, surely the production issues are cause by high peak loads like a new chapter of the latest series or whatever.

I'm all over the place, I just don't like netflix is what I'm saying

Re: We built a self-healing system to survive a concurrency bug at Netflix

#112

Post by romanobro56 »

TZubiri wrote: post #111

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like wh…

[flagged]

Re: We built a self-healing system to survive a concurrency bug at Netflix

#113

Post by PittleyDunkin »

TZubiri wrote: post #111

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like wh…

> But as time goes by I just ask, all this work and costs and complexity, to serve files?

You could say the same thing about the entire web.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#114

Post by eschneider »

DrBazza wrote: post #79
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

Sounds like process-level garbage collection. Just kill it and restart. Which also sound like the apocryphal tale about the leaky code and the missile. "This sparked and interesting memory for me. I was once working with a customer who was producing on-board software for a missile. In my analysis of the code, I pointed out that they had a number of problems with storage leaks. Imagine my surprise when the customers c…

At least with the missile case, someone _did the analysis and knows exactly what's wrong_ before deciding the "solution" was letting the resources leak. That's fine.

What always bothers me, is when (note, I'm not saying this is the case for the grandparent comment, but it's implied) people don't understand what exactly is broken, but just reboot every so often to fix things. :0

For a lot of bugs, there's often the component you see (like the obvious resource leak) combined with subtle problems you don't see (data corruption, perhaps?) and you won't really know until the problem is tracked down.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#115

Post by nicce »

PittleyDunkin wrote: post #113
TZubiri wrote: post #111

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like wh…

> But as time goes by I just ask, all this work and costs and complexity, to serve files? You could say the same thing about the entire web.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#116

Post by 4star3star »

JonChesterfield wrote: post #58

Title is grossly misleading. That Netflix had already built a self-healing system means they were able to handle a memory leak by killing random servers faster than memory was leaking. This post isn't about how they've managed that, it's just showing off that their existing system is robust enough that you can do hacks like this to it.

Your take is much different than mine. The issue was a practical one of sparing people from working too much over one weekend since the bug would have to wait until Monday, and the author willingly described the solution as the worst.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#117

Post by nesarkvechnep »

cpursley wrote: post #71
conradfr wrote: post #28

I have a project where one function (reading metadata from an Icecast stream [0]) was causing a memory leak and ultimately consuming all of it. I don't remember all the details but I've still not be able to find the bug. But this being in Elixir I "fixed it" with Task, TaskSupervisor and try/catch/rescue. Not really a win but it is still running fine to this day. [0] https://github.com/conradfr/ProgRadio/blob/1fa12ca…

Half of hn posts are people showing off things where they spent a herculean amount of effort reinventing something that elixir/erlang has had solved 30+ years already.

Some are even proud of their ignorance and belittle Erlang and Elixir.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#118

Post by TZubiri »

romanobro56 wrote: post #112
TZubiri wrote: post #111

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like wh…

[flagged]

Rizz skibidi bro

Re: We built a self-healing system to survive a concurrency bug at Netflix

#119

Post by TZubiri »

nicce wrote: post #115
PittleyDunkin wrote: post #113

Earlier quoted context omitted.

> But as time goes by I just ask, all this work and costs and complexity, to serve files? You could say the same thing about the entire web.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

I thought about the complexity in terms of compute, but I guess if there's no input then there's no compute possible, as all functions are idempotent and static. At the very least their results are cacheable, or the input is centralized (admins/show producers)

Re: We built a self-healing system to survive a concurrency bug at Netflix

#120

Post by xxs »

vladak wrote: post #103
xxs wrote: post #18

They were just lucky not to have data corruption due to concurrency issue, and the manifestation was infinite get. Overall if you can randomly "kill -9", the case is rather trivial. Likely replacing HashMap with CHM would not solve the concurrency issue either, but it'd prevent an infinite loop. (Edit) It appear that part is just wrong: "some calls to ConcurrentHashMap.get() seemed to be running infinitely." <-- it's…

yep, the link in the "some calls to ConcurrentHashMap.get() seemed to be running infinitely." sentence points to HashMap.html#get(java.lang.Object)

I have seen that part myself (infinite loops), also I have quite extensive experience with CHM (and HashMap).

Overall such a mistake alone undermines the effort/article.

Post reply on HN