Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta patches Linux at hyperscale

thenewstack.io

Post reply on HN
Print view View original

21–30 of 114 posts

Re: How Meta patches Linux at hyperscale

#21

Post by zug_zug »

tester756 wrote: post #16
zug_zug wrote: post #9

Earlier quoted context omitted.

> In computing, hyperscale is the ability of an architecture to scale appropriately as increased demand is added to the system. So yeah just scaling. I agree I've never heard the word "hyperscale" before and don't think we need that extra intensifier for a well-understood idea.

1x, 10x, 1000x are all "scales", yet problems may be a little bit different at each of them

I'm just quoting the wikipedia definiton of hyperscale. Nowhere in there does it say anything about 1000x, probably because that is an ill-defined concept.

1000x what? Today's computers are a 1000x the ones from the 90s, should we call them all hypercomputers? Pretty much any startup can boot 20,000 nodes on aws, are they all hyperstartups hyperscaling?

Seems like marketing nonsense.

Re: How Meta patches Linux at hyperscale

#22

Post by faitswulff »

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did.

Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

Re: How Meta patches Linux at hyperscale

#24

Post by kreeben »

zug_zug wrote: post #21
tester756 wrote: post #16

Earlier quoted context omitted.

1x, 10x, 1000x are all "scales", yet problems may be a little bit different at each of them

I'm just quoting the wikipedia definiton of hyperscale. Nowhere in there does it say anything about 1000x, probably because that is an ill-defined concept. 1000x what? Today's computers are a 1000x the ones from the 90s, should we call them all hypercomputers? Pretty much any startup can boot 20,000 nodes on aws, are they all hyperstartups hyperscaling? Seems like marketing nonsense.

I'm guessing you're mad about the term but not the concept? And that you do agree, that scaling from 1 to 10 is not the same animal as scaling from 10 to 100? So why, then, call both animals "plain and simple scaling"?

Re: How Meta patches Linux at hyperscale

#25

Post by baby_souffle »

faitswulff wrote: post #22

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

> Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

SO. Much. This.

Worked at more than a few places where the stack had almost as many layers as it had engineers "because this is how we did it at FAANG..."

Right, and those places also had a few orders of magnitude more engineers on staff to support it all. We do one hundredth of the things FAANG does and we have less than 50 _total_ people in the company; the simpler the stack, the better.

Re: How Meta patches Linux at hyperscale

#26

Post by WookieRushing »

hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

Why do you think its easy?

There's a lot of systems where you can easily take down some hosts, but taking down more than N% at a time causes issues. If your fleet is large enough then you are limited by the largest set of hosts where you can only take N% down at a time. Now you could say keep the sets of hosts small or N% large. But that can cause other issues as you typically lose efficiency or zonal outage protection.

A solution to this could be VM live migration or something similar. This breaks down for storage systems where you can't just migrate those disks virtually since they're physical disks or places that don't use VMs.

Re: How Meta patches Linux at hyperscale

#27

Post by TheAceOfHearts »

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit.

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did.

Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I can't really fathom the complexity of managing millions of servers though.

Re: How Meta patches Linux at hyperscale

#28

Post by dboreham »

zug_zug wrote: post #9
tester756 wrote: post #8

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Hyperscale_computing

> In computing, hyperscale is the ability of an architecture to scale appropriately as increased demand is added to the system. So yeah just scaling. I agree I've never heard the word "hyperscale" before and don't think we need that extra intensifier for a well-understood idea.

The word hyperscale afaik was coined by Wall St people as a collective noun for FB, Google, MS, et al, in the context of their in-house data center operations.

Re: How Meta patches Linux at hyperscale

#30

Post by agrajag »

phyzome wrote: post #20
ravenstine wrote: post #4

Never heard of this "hyperscale" concept before. How is this any different from... scaling?

Yeah, and this isn't a problem that interacts with scale. If you can patch 100 servers with an automated method you can patch 1,000,000 of them.

This is absolutely a problem that interacts with scale. With 1M servers you’re almost certainly dealing with hundreds of service owners, and some of those are going to need additional features you don’t have to worry about with 100 servers. Some examples are databases with graceful failover, long running AI model training jobs, or distributed databases like etc where you have to be mindful about how many can be down at a time.

It’s not 10,000x harder to patch that 10,000x more machines, but it’s not 1x either. Easily 10-20x harder, if not more.

Post reply on HN