Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

151–160 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#151

Post by uvdn7 »

posix_compliant wrote: post #126

What's neat is that this is a differential equation. If you kill 5% of instances each hour, the reduction in bad instances is proportional to the current number of instances. i.e. if bad(t) = fraction of bad instances at time t and bad(0) = 0 then d(bad(t))/dt = -0.05 * bad(t) + 0.01 * (1 - bad(t)) so bad(t) = 0.166667 - 0.166667 e^(-0.06 t) Which looks a mighty lot like the graph of bad instances in the blog post.

Love it! I wonder if the team knew this explicitly or intuitively when they deployed the strategy.

> We created a rule in our central monitoring and alerting system to randomly kill a few instances every 15 minutes. Every killed instance would be replaced with a healthy, fresh one.

It doesn't look like they worked out the numbers ahead of the time.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#152

Post by lanstin »

pmarreck wrote: post #100
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

Heroku reboots servers every night no matter what stack is running on them. Same idea. The problem is that you merely borrowed yourself some time. As time goes on, more inefficiencies/bugs of this nature will creep in unnoticed, some will perhaps silently corrupt data before it is noticed (!), and it will be vastly more difficult at that point to troubleshoot 10 bugs of varying degrees of severity and frequency all h…

Also, it means you are married to the reboot process. If you loose control of your memory management process too much, you'll never be able to fix it absent a complete rewrite. I worked at a place that had a lot of (c++) CGI programs with a shocking level of disregard for freeing memory, but that was ok because when the CGI request was over the process restarted. But then they reused that same code in SOA/long lived services, but they could never have one worker process handle more than 10 requests due to memory leaks (and inability to re-initialize all the memory used in a request). So they could never use in-process caching or any sort of optimization that long-lived processes could enable.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#153

Post by rtkwe »

cruffle_duffle wrote: post #147
pjdesno wrote: post #134

Vaguely related anecdote: 30 years ago or so I worked at a tiny networking company where several coworkers came from a small company (call it C) that made AppleTalk routers. They recounted being puzzled that their competitor (company S) had a reputation for having a rock-solid product, but when they got it into the lab they found their competitor's product crashed maybe 10 times more often than their own. It turned o…

Seems like the next priority would be to make your product reboot just as fast if not faster then theirs.

Clearly but maybe the thing that makes your product crash less makes it take longer to reboot.

Also the story isn't that they couldn't just that they were measuring the actual failure rate not the effective failure rate because the device could recover faster than the failure caused actual issues.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#154

Post by otterley »

jldugger wrote: post #150
otterley wrote: post #135

> It was Friday afternoon > Rolling back was cumbersome It's a fundamental principle of modern DevOps practice that rollbacks should be quick and easy, done immediately when you notice a production regression, and ideally automated. And at Netflix's scale, one would have wanted this rollout to be done in waves to minimize risk. Apparently this happened back in 2021. Did the team investigate later why you couldn't do…

>It's a fundamental principle of modern DevOps practice that rollbacks should be quick and easy Then DevOps principles are in conflict with reality.

Go on...

Re: We built a self-healing system to survive a concurrency bug at Netflix

#155

Post by PittleyDunkin »

nicce wrote: post #115
PittleyDunkin wrote: post #113

Earlier quoted context omitted.

> But as time goes by I just ask, all this work and costs and complexity, to serve files? You could say the same thing about the entire web.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

Is it? It's pretty rare to download assets from servers that you're uploading to. Sometimes you have truly interactive app servers but that's a pretty small percentage of web traffic. Shared state is not the typical problem to solve on the internet, though it is a popular one to discuss.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#156

Post by PittleyDunkin »

nicce wrote: post #140
jacksontheel wrote: post #137

Earlier quoted context omitted.

Every time you like/dislike/watchlist a movie you're posting data. When you're watching a movie your progress is constantly updated, posting data. Simple stuff but there's possibly hundreds of thousands of concurrent users doing that at any given moment.

Yes, but it is still counts only a fraction of the purpose of their infrastructure. There are no hard global real-time sync requirements. > When you're watching a movie your progress is constantly updated, posting data This can be implemented on server side and with read requests only. A proper comparison would be YouTube where people upload videos and comment stuff in real-time.

> A proper comparison would be YouTube where people upload videos and comment stuff in real-time.

Even in this one sentence you're conflating two types of interaction. Surely downloading videos is yet a third, and possibly the rest of the assets on the site a fourth.

Why not just say the exact problem you think is worth of discussion with your full chest if you so clearly have one in mind?

Re: We built a self-healing system to survive a concurrency bug at Netflix

#157

Post by kevin_nisbet »

pjdesno wrote: post #134

Vaguely related anecdote: 30 years ago or so I worked at a tiny networking company where several coworkers came from a small company (call it C) that made AppleTalk routers. They recounted being puzzled that their competitor (company S) had a reputation for having a rock-solid product, but when they got it into the lab they found their competitor's product crashed maybe 10 times more often than their own. It turned o…

This is along the lines of how one of the wireless telecom products I really liked worked.

Each running process had a backup on another blade in the chassis. All internal state was replicated. And the process was written in a crash only fashion, anything unexpected happened and the process would just minicore and exit.

One day I think I noticed that we had over a hundred thousand crashes in the previous 24 hours, but no one complained and we just sent over the minicores to the devs and got them fixed. In theory some users would be impacted that were triggering the crashes, their devices might have a glitch and need to re-associate with the network, but the crashes caused no widespread impacts in that case.

To this day I'm a fan of crash only software as a philosophy, even though I haven't had the opportunity to implement it in the software I work on.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#158

Post by rikthevik »

nesarkvechnep wrote: post #117
cpursley wrote: post #71

Earlier quoted context omitted.

Half of hn posts are people showing off things where they spent a herculean amount of effort reinventing something that elixir/erlang has had solved 30+ years already.

Some are even proud of their ignorance and belittle Erlang and Elixir.

I'm fine with it.

If people want to belittle something, either we aren't trying to solve the same problem (sure) or they're actively turning people away from what could be a serious advantage (more for me!)

If the cost of switching wasn't so high, I'd love to write Elixir all day. It's a joy.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#159

Post by nicce »

PittleyDunkin wrote: post #155
nicce wrote: post #115

Earlier quoted context omitted.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

Is it? It's pretty rare to download assets from servers that you're uploading to. Sometimes you have truly interactive app servers but that's a pretty small percentage of web traffic. Shared state is not the typical problem to solve on the internet, though it is a popular one to discuss.

Whatever your service is, usually the database is the bottleneck. The database limits the latency, scaling and availability.

Of course, how much, depends on the service. Particularly, how much concurrent writing is happening, and do you need to update this state globally, in real-time as result of this writing. Also, is local caching happening and do you need to invalidate the cache as well as a result of this writing.

The most of the relevant problems disappear, if you can just replicate most of the data without worrying that someone is updating it, and you also don't have cache invalidation issues. No race conditions. No real-time replication issues.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#160

Post by mech422 »

zoky wrote: post #74
crabbone wrote: post #69

Earlier quoted context omitted.

Oh no it isn't. Garbage collector needs to prove that what's being collected is garbage. If objects get collected because of an error... that's not really how you want GC to work. If you are looking for an apt metaphor, Stalin sort might be more in line with what's going on here. Or maybe "ostrich algorithm".

I think it’s more like Tech Support Sort, as in “Try turning it off and on again and see if it’s sorted”.

LOL - I like that one! :-)
Post reply on HN