Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

131–140 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#131

Post by resize2996 »

keeganpoppen wrote: post #33
anal_reactor wrote: post #27

Earlier quoted context omitted.

I've realized that majority of engineers have no critical thinking, and are unable to see things beyond their domain of speciality. Arguments like "even when accounting for potential incident, your solution is more expensive, while our main goal is making money" almost never work, and I've been in countless discussions where some random document with "best practices", whatever they are supposed to be, was treated lik…

that's because the judge(s) and executioner(s) aren't engineers, and the jury is not of their peers. and for the record i have a hard time faulting the non-engineers above so-described... they are just grasping for things they can understand and have input on. who wouldn't want that? it's not at all reasonable for the keepers of the pursestrings to expect a certain amount of genuflection by way of self-justification.…

I dig your vibe. whaddya working on these days?

Re: We built a self-healing system to survive a concurrency bug at Netflix

#132

Post by mhawthorne »

gorkempacaci wrote: post #19
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

How about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)

[dead]

Re: We built a self-healing system to survive a concurrency bug at Netflix

#133

Post by xeromal »

chronid wrote: post #82
whatever1 wrote: post #62

Earlier quoted context omitted.

I think this is a prime example of why the cloud won. You don’t need wizards in your team anymore. Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

This has been done forever. Ops team had cronjobs to restart misbehaving applications out of business hours since before I started working. In a previous job, the solution for disks being full on a VM on-prem (no, not databases) was an automatic reimage. I've seen scheduled index rebuilds on Oracle. The list goes on.

Rebooting Windows IIS instances every night has been a mainstay for most of my career. haha

Re: We built a self-healing system to survive a concurrency bug at Netflix

#134

Post by pjdesno »

Vaguely related anecdote:

30 years ago or so I worked at a tiny networking company where several coworkers came from a small company (call it C) that made AppleTalk routers. They recounted being puzzled that their competitor (company S) had a reputation for having a rock-solid product, but when they got it into the lab they found their competitor's product crashed maybe 10 times more often than their own.

It turned out that the competing device could reboot faster than the end-to-end connection timeout in the higher-level protocol, so in practice failures were invisible. Their router, on the other hand, took long enough to reboot that your print job or file server copy would fail. It was as simple as that, and in practice the other product was rock-solid and theirs wasn't.

(This is a fairly accurate summary of what I was told, but there's a chance my coworkers were totally wrong. The conclusion still stands, I think - fast restarts can save your ass.)

Re: We built a self-healing system to survive a concurrency bug at Netflix

#135

Post by otterley »

> It was Friday afternoon

> Rolling back was cumbersome

It's a fundamental principle of modern DevOps practice that rollbacks should be quick and easy, done immediately when you notice a production regression, and ideally automated. And at Netflix's scale, one would have wanted this rollout to be done in waves to minimize risk.

Apparently this happened back in 2021. Did the team investigate later why you couldn't do this, and address it?

Re: We built a self-healing system to survive a concurrency bug at Netflix

#136

Post by mhawthorne »

dhruvrrp wrote: post #10

Interesting read, the fix seems to be straightforward, but I'd have a few more questions if I was trying to do something similar. Is software deployed regularly on this cluster? Does that deployment happen faster than the rate at which they were losing CPUs? Why not just periodically force a deployment, given it's a repeated process that probably already happens frequently. What happens to the clients trying to conne…

[dead]

Re: We built a self-healing system to survive a concurrency bug at Netflix

#137

Post by jacksontheel »

nicce wrote: post #115
PittleyDunkin wrote: post #113

Earlier quoted context omitted.

> But as time goes by I just ask, all this work and costs and complexity, to serve files? You could say the same thing about the entire web.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

Every time you like/dislike/watchlist a movie you're posting data. When you're watching a movie your progress is constantly updated, posting data. Simple stuff but there's possibly hundreds of thousands of concurrent users doing that at any given moment.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#138

Post by __turbobrew__ »

> Could we roll back? Not easily. I can’t recall why

I can appreciate the hack to deal with this (I actually came up with the same solution in my head as reading) but if you cannot rollback and you cannot roll forward you are stuck in a special purgatory of CD hell that you should be spending every moment of time getting out of before doing anything else.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#139

Post by bobdvb »

TZubiri wrote: post #111

Netflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like wh…

Netflix has done massive amounts of work on BSD to improve it's network throughput, that's part of them enabling their file delivery from their CDN appliances. https://people.freebsd.org/~gallatin/talks/euro2022.pdf

They've also contributed significantly to open source tools for video processing, one of the biggest things that stands out is probably their VMAF tool for quantifying perceptual quality in video. It's probably the best open source tool for measuring video quality out there right now.

It's also absolutely true that in any streaming service, the orchestration, account management, billing and catalogue components are waaaay more complex than actually delivering video on-demand. To counter one thing you've said: mouse movement... most viewing of premium content isn't done on web or even mobile devices. Most viewing time of paid content is done on a TV, where you're not measuring focus. But that's just a piece of trivia.

As you said, you just don't like them, but they've done a lot for the open source community and that should be understood.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#140

Post by nicce »

jacksontheel wrote: post #137
nicce wrote: post #115

Earlier quoted context omitted.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

Every time you like/dislike/watchlist a movie you're posting data. When you're watching a movie your progress is constantly updated, posting data. Simple stuff but there's possibly hundreds of thousands of concurrent users doing that at any given moment.

Yes, but it is still counts only a fraction of the purpose of their infrastructure. There are no hard global real-time sync requirements.

> When you're watching a movie your progress is constantly updated, posting data

This can be implemented on server side and with read requests only.

A proper comparison would be YouTube where people upload videos and comment stuff in real-time.

Post reply on HN