Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta patches Linux at hyperscale

thenewstack.io

Post reply on HN
Print view View original

41–50 of 114 posts

Re: How Meta patches Linux at hyperscale

#41

Post by fragmede »

ChatGTP wrote: post #37
TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

[flagged]

Ads are just what pays the bills. The cluster is also used for all sorts of useful things, like mass communication, especially during emergencies, support groups, and providing information to help manage a pandemic.

Software engineers aren't nuclear physicists or machinists. I'm not sure how you want to use them to build nuclear reactors.

Re: How Meta patches Linux at hyperscale

#42

Post by [deleted] »

hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

[deleted]

Re: How Meta patches Linux at hyperscale

#43

Post by nickysielicki »

faitswulff wrote: post #22

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

This sort of criticism gets repeated all the time ("Google designs for Google scale, but you're not Google, so don't use kubernetes!"), and sometimes it's fair, but this doesn't really make sense to me on this particular article.

If the infrastructure exists within your org's distribution of choice to do this, it's basically all upside. On AL2023, you just do:

`sudo dnf install -y kpatch-dnf kpatch-runtime`

`sudo dnf kernel-livepatch -y auto`

`sudo systemctl enable --now kpatch.service`

Super simple, one less thing to worry about.

Re: How Meta patches Linux at hyperscale

#44

Post by sweettea »

TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

https://youtube.com/watch?v=ILTqn1EYIXQ is the original talk, which says it takes 4 days to deploy a KLP to the whole fleet.

Re: How Meta patches Linux at hyperscale

#45

Post by sweettea »

ChatGTP wrote: post #37
TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

[flagged]

Ads act as a recommendation engine, helping people find solutions to their problems sometimes before they ever realize there is a solution.

Re: How Meta patches Linux at hyperscale

#46

Post by ChatGTP »

sweettea wrote: post #45
ChatGTP wrote: post #37

Earlier quoted context omitted.

[flagged]

Ads act as a recommendation engine, helping people find solutions to their problems sometimes before they ever realize there is a solution.

lol common now. It’s to make Facebook a bunch of money. Not for tell the future.

I use facebooks products and it’s never helped me or anyone I know in the way you describe.

Re: How Meta patches Linux at hyperscale

#47

Post by ChatGTP »

phyrex wrote: post #40
ChatGTP wrote: post #37

Earlier quoted context omitted.

[flagged]

It’s a very reductive take. Meta is helping a lot of people get and keep in touch for free (like my family is all over the place and we use WhatsApp extensively to communicate). There’s clearly a demand for this, and doing it for the price of a couple of ads doesn’t seem sad to me

No, the users are the products. It’s not free.

Re: How Meta patches Linux at hyperscale

#48

Post by phyrex »

mc32 wrote: post #5
ravenstine wrote: post #4

Never heard of this "hyperscale" concept before. How is this any different from... scaling?

Supermarkets and super cars aren't enough. We need hypermarkets and hyper cars to be current. So now we can’t just scale, I guess that’s for trucks, so to be current you need hyperscale.

https://en.m.wikipedia.org/wiki/Hypermarket ¯\_(ツ)_/¯

Re: How Meta patches Linux at hyperscale

#49

Post by phyrex »

ChatGTP wrote: post #47
phyrex wrote: post #40

Earlier quoted context omitted.

It’s a very reductive take. Meta is helping a lot of people get and keep in touch for free (like my family is all over the place and we use WhatsApp extensively to communicate). There’s clearly a demand for this, and doing it for the price of a couple of ads doesn’t seem sad to me

No, the users are the products. It’s not free.

You’re not paying them out of your own pocket. I’ll happily use a better word for that if you have one.

Re: How Meta patches Linux at hyperscale

#50

Post by mypalmike »

hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

> fixing their system for taking hosts in and out of service, so that it's extremely robust and reliable would likely pay big dividends in reliability.

Facebook hosts can be robustly cycled. Of course. They've been doing this stuff for years. They've figured it out. That's not the issue.

Scaling up brings about new problems. This article specifically mentions the 45 day rolling restart issue. That's not an issue when you have 1000s of servers. It's one that shows up a couple of orders of magnitude later.

So you either solve the problem with a hack like kernel patching or you work to reduce restart times (drain + shutdown + OS restart + process initialization across every service). Get those restart times down 50% (good luck accomplishing that) and congrats, you're down to maybe a 25 day rolling restart, which is still quite a problem.

Post reply on HN