Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta patches Linux at hyperscale

thenewstack.io

Post reply on HN
Print view View original

31–40 of 114 posts

Re: How Meta patches Linux at hyperscale

#31

Post by comprev »

ravenstine wrote: post #4

Never heard of this "hyperscale" concept before. How is this any different from... scaling?

The "H" in ADHD stands for "hyperactivity". This implies _significantly_ more activity than the norm to a point where they really stand out.

In sci-fi a spaceship travelling at "hyperspeed" was perceived to be so much faster than anything known to man that it would be difficult to comprehend.

The performance and cost of a hypercar compared to the average family car is sometimes difficult to understand too. The average person would have to work (potentially) hundreds of years to afford a €10M hypercar.

"Hyperscale" is so large that even us working in tech have difficulty grasping it because we have nothing tangible to compare it to. A million servers is bind boggling to me even with the "cattle not pets" mindset

Re: How Meta patches Linux at hyperscale

#32

Post by fragmede »

metadat wrote: post #13
fragmede wrote: post #2

Ksplice is the original live patching technology that got bought by Oracle and was later extended to user space programs while I worked there. It's a really neat technology that isn't made obsolete by the move to cloud, since you still don't want to have to restart the whole fleet at scale.

What's the live app patcher called?

I have objections to the name, but it goes under Ksplice for User Space.

https://blogs.oracle.com/virtualization/post/ksplice-zero-do...

Re: How Meta patches Linux at hyperscale

#33

Post by zug_zug »

kreeben wrote: post #24
zug_zug wrote: post #21

Earlier quoted context omitted.

I'm just quoting the wikipedia definiton of hyperscale. Nowhere in there does it say anything about 1000x, probably because that is an ill-defined concept. 1000x what? Today's computers are a 1000x the ones from the 90s, should we call them all hypercomputers? Pretty much any startup can boot 20,000 nodes on aws, are they all hyperstartups hyperscaling? Seems like marketing nonsense.

I'm guessing you're mad about the term but not the concept? And that you do agree, that scaling from 1 to 10 is not the same animal as scaling from 10 to 100? So why, then, call both animals "plain and simple scaling"?

> And that you do agree, that scaling from 1 to 10 is not the same animal as scaling from 10 to 100? So why, then, call both animals "plain and simple scaling"?

So you think we need 4 different words for scaling 1-10, 10-100, 100-1000....?

Cut it out, you know it's just marketing hype. Not everything needs its own word.

Re: How Meta patches Linux at hyperscale

#34

Post by natbennett »

agrajag wrote: post #30
phyzome wrote: post #20

Earlier quoted context omitted.

Yeah, and this isn't a problem that interacts with scale. If you can patch 100 servers with an automated method you can patch 1,000,000 of them.

This is absolutely a problem that interacts with scale. With 1M servers you’re almost certainly dealing with hundreds of service owners, and some of those are going to need additional features you don’t have to worry about with 100 servers. Some examples are databases with graceful failover, long running AI model training jobs, or distributed databases like etc where you have to be mindful about how many can be down…

And at a certain scale patches will come out faster than you can deploy them!

Re: How Meta patches Linux at hyperscale

#35

Post by natbennett »

hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

The work that makes cycling hosts in and out easy is itself hard.

I agree that it’s the right thing to do but it’s hard.

Re: How Meta patches Linux at hyperscale

#36

Post by influx »

faitswulff wrote: post #22

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

Likewise at Meta there's a bunch of cargo-culting from Google.

Re: How Meta patches Linux at hyperscale

#37

Post by ChatGTP »

TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

[flagged]

Re: How Meta patches Linux at hyperscale

#38

Post by phyrex »

influx wrote: post #36
faitswulff wrote: post #22

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

Likewise at Meta there's a bunch of cargo-culting from Google.

Could you expand please?

Re: How Meta patches Linux at hyperscale

#39

Post by fragmede »

hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

It's not a bullshit reason. You can put all the lipstick you want on the pig and put software all around it to make it all sorts of easy, but at the end of the day, having to reboot is a stop-the-(machine's)-world situation. Not having to do that is just better. Even if it doesn't work 100% of the time, that's still better than having to reboot the whole fleet. 45 days to reboot the whole fleet!

Throwing FUD and saying disaster is looming because its scary computer magic (out of MIT) was a scare tactic RedHat used to throw around about Oracle/Ksplice until they developed their own (Kpatch), then suddenly their sales team had to backtrack and say actually hot patching is good and can be trusted. I'm not saying it's not risky or dangerous, it's operating in kernel space, but that's why they pay really smart people to be careful when doing it, and not digital equivalent of a plumber who can't do more than glue libraries together.

A better understanding of the underlying technology so it's less magic might assuage your fear of it, but thinking Facebook is so dysfunctional that they haven't already made it easier to reboot is to misunderstand the problem at hand.

Re: How Meta patches Linux at hyperscale

#40

Post by phyrex »

ChatGTP wrote: post #37
TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

[flagged]

It’s a very reductive take. Meta is helping a lot of people get and keep in touch for free (like my family is all over the place and we use WhatsApp extensively to communicate). There’s clearly a demand for this, and doing it for the price of a couple of ads doesn’t seem sad to me
Post reply on HN