We built a self-healing system to survive a concurrency bug at Netflix

Post by mech422 » Wed, Nov 13, 2024, 7:33 PM UTC

raverbashing wrote: post #42

keeganpoppen wrote: post #34

Earlier quoted context omitted.

"certain kind" of discipline, indeed... not the good kind. and while your comment goes to great pains to highlight how that particular God is dead (and i agree, for the record), the God of Quality (the one that Pirsig goes to great lengths to not really define) toward which the engineer's heart of heart prays that lives within us all is... unimpressed, to say the least.

Sure, you worship the God of Quality until you realize that memory leak is being caused by a 3rd party library (extra annoying when you could have solved it yourself) or a quirky stdlib implementation Then you realize it's a paper idol and the best you can do is suck less than the average. Thanks for playing Wing Commander!

Post by tommygpt » Wed, Nov 13, 2024, 7:42 PM UTC

PittleyDunkin wrote: post #155

nicce wrote: post #115

Earlier quoted context omitted.

Not really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.

Is it? It's pretty rare to download assets from servers that you're uploading to. Sometimes you have truly interactive app servers but that's a pretty small percentage of web traffic. Shared state is not the typical problem to solve on the internet, though it is a popular one to discuss.

Post by pmarreck » Wed, Nov 13, 2024, 8:21 PM UTC

lanstin wrote: post #152

pmarreck wrote: post #100

Earlier quoted context omitted.

Heroku reboots servers every night no matter what stack is running on them. Same idea. The problem is that you merely borrowed yourself some time. As time goes on, more inefficiencies/bugs of this nature will creep in unnoticed, some will perhaps silently corrupt data before it is noticed (!), and it will be vastly more difficult at that point to troubleshoot 10 bugs of varying degrees of severity and frequency all h…

Also, it means you are married to the reboot process. If you loose control of your memory management process too much, you'll never be able to fix it absent a complete rewrite. I worked at a place that had a lot of (c++) CGI programs with a shocking level of disregard for freeing memory, but that was ok because when the CGI request was over the process restarted. But then they reused that same code in SOA/long lived…

Post by nicce » Wed, Nov 13, 2024, 10:53 PM UTC

PittleyDunkin wrote: post #162

nicce wrote: post #159

Earlier quoted context omitted.

Whatever your service is, usually the database is the bottleneck. The database limits the latency, scaling and availability. Of course, how much, depends on the service. Particularly, how much concurrent writing is happening, and do you need to update this state globally, in real-time as result of this writing. Also, is local caching happening and do you need to invalidate the cache as well as a result of this writin…

> Whatever your service is, usually the database is the bottleneck. The database limits the latency, scaling and availability. Database-driven traffic is still a tiny percentage of internet traffic. It's harder to tell these days with encryption but on any given page-load on any project I've worked on, most of the traffic is in assets, not application data. Now, latency might be a different issue, but it seems ridicu…

Post by andrewf » Thu, Nov 14, 2024, 1:27 AM UTC

lanstin wrote: post #152

pmarreck wrote: post #100

Earlier quoted context omitted.

Heroku reboots servers every night no matter what stack is running on them. Same idea. The problem is that you merely borrowed yourself some time. As time goes on, more inefficiencies/bugs of this nature will creep in unnoticed, some will perhaps silently corrupt data before it is noticed (!), and it will be vastly more difficult at that point to troubleshoot 10 bugs of varying degrees of severity and frequency all h…

Also, it means you are married to the reboot process. If you loose control of your memory management process too much, you'll never be able to fix it absent a complete rewrite. I worked at a place that had a lot of (c++) CGI programs with a shocking level of disregard for freeing memory, but that was ok because when the CGI request was over the process restarted. But then they reused that same code in SOA/long lived…