Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a self-healing system to survive a concurrency bug at Netflix

pushtoprod.substack.com

Post reply on HN
Print view View original

91–100 of 184 posts

Re: We built a self-healing system to survive a concurrency bug at Netflix

#91

Post by braggerxyz »

chronid wrote: post #82
whatever1 wrote: post #62

Earlier quoted context omitted.

I think this is a prime example of why the cloud won. You don’t need wizards in your team anymore. Something seems off in the instance? Just nuke it and spin up a new one. Let the system debugging for the Amazon folks.

This has been done forever. Ops team had cronjobs to restart misbehaving applications out of business hours since before I started working. In a previous job, the solution for disks being full on a VM on-prem (no, not databases) was an automatic reimage. I've seen scheduled index rebuilds on Oracle. The list goes on.

> I've seen scheduled index rebuilds on Oracle

If you do look into the Oracle dba handbook, scheduled index rebuilds are somewhat recommended. We do it on weekends on our Oracle instances. Otherwise you will encounter severe performance degredation in tables where data is inserted and deleted at high throughput thus leading to fragmented indexes. And since Oracle 12g with ONLINE REBUILD this is no problem anymore even at peak hours.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#92

Post by braggerxyz »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

> It's shockingly stable. So much so that resolving the root cause isn't considered a priority and so we've had this running for months.

The trick is to not tell your manager that your bandaid works so well, but that it barely keeps the system alive and you need to introduce a proper fix. Been doing this for the last 10 years and we got our system so stable that I haven't had a midnight call in the last two years.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#93

Post by Retric »

netdevnet wrote: post #54
kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

> It's shockingly stable. So much so that resolving the root cause isn't considered a priority and so we've had this running for months. I don't know why my senses tell me that this is wrong even if you can afford it

> I don't know why my senses tell me that this is wrong

The fix is also hiding other issues that show up. So it degrades over time and eventually you’re stuck trying to solve multiple problems at the same time.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#94

Post by Sl1mb0 »

louison11 wrote: post #85

You gotta pick your battles. Part of being in a startup is to be comfortable with quick and dirty when necessary. It’s when things get bigger, too corporate and slow that companies stop moving fast.

We are talking about Netflix. You know, the 'N' in FAANG/MAANG or whatever.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#95

Post by crabbone »

nukethegrbj wrote: post #83
crabbone wrote: post #69

Earlier quoted context omitted.

Oh no it isn't. Garbage collector needs to prove that what's being collected is garbage. If objects get collected because of an error... that's not really how you want GC to work. If you are looking for an apt metaphor, Stalin sort might be more in line with what's going on here. Or maybe "ostrich algorithm".

>Garbage collector needs to prove that what's being collected is garbage Some collectors may need to do this, but there are several collectors that don't. EpsilonGC is a prime example of a GC that doesen't need to prove anything

EpsilonGC is a GC in the same sense as a suitable-size stick is a fully automatic rifle when you hold it to your shoulder and say pew-pew...

I mean, I interpret your comment to be a joke, but you could've made it a bit more obvious for people not familiar with the latest fancy in Java world.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#96

Post by [deleted] »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

[deleted]

Re: We built a self-healing system to survive a concurrency bug at Netflix

#97

Post by morning-coffee »

nicman23 wrote: post #15
est wrote: post #13

Reminds me of the famous quote by Rasmus Lerdorf, creator of PHP > I’m not a real programmer. I throw together things until it works then I move on. The real programmers will say “Yeah it works but you’re leaking memory everywhere. Perhaps we should fix that.” I’ll just restart Apache every 10 requests.

i ll argue that doing the restart is more important until someone else finds the leak

Also uttered by others who thought borrowing money was more important until they could figure out a way to control spending.

Re: We built a self-healing system to survive a concurrency bug at Netflix

#98

Post by pmarreck »

I had to deal with a concurrency bug in Ruby once and it was so bad* that it pushed me into Elixir, which makes the vast majority of concurrency bugs impossible at the language-design level, thus enabling more sanity.

Ingeniously simple solution for this particular bug though.

*as I recall, it had to do with merging a regular Hash in the ENV with a HashWithIndifferentAccess, which as it turns out was ill-conceived at the time and had undefined corner cases (example: what should happen when you merge a regular Hash containing either a string or symbol key (or both) into a HashWithIndifferentAccess containing the same key but internally only represented as a string? Which takes precedence was undefined at the time.)

Re: We built a self-healing system to survive a concurrency bug at Netflix

#99

Post by pmarreck »

btbuildem wrote: post #89

On a long enough timescale, everything eventually converges to Erlang

Hah, hinted at that in my comment: https://news.ycombinator.com/item?id=42126301

It really is a fundamental advantage against the worst kinds of this category of bug

Re: We built a self-healing system to survive a concurrency bug at Netflix

#100

Post by pmarreck »

kenhwang wrote: post #16

My workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours.…

Heroku reboots servers every night no matter what stack is running on them. Same idea.

The problem is that you merely borrowed yourself some time. As time goes on, more inefficiencies/bugs of this nature will creep in unnoticed, some will perhaps silently corrupt data before it is noticed (!), and it will be vastly more difficult at that point to troubleshoot 10 bugs of varying degrees of severity and frequency all happening at the same time causing you to have to reboot said servers at faster and faster intervals which simultaneously makes it harder to diagnose them individually.

> It's shockingly stable.

Well of course it is. You're "turning it off and then on again," the classic way to return to a known-good state. It is not a root-cause fix though, it is a band-aid.

Post reply on HN