Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta patches Linux at hyperscale

thenewstack.io

Post reply on HN
Print view View original

61–70 of 114 posts

Re: How Meta patches Linux at hyperscale

#61

Post by hellozomo »

mypalmike wrote: post #50
hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

> fixing their system for taking hosts in and out of service, so that it's extremely robust and reliable would likely pay big dividends in reliability. Facebook hosts can be robustly cycled. Of course. They've been doing this stuff for years. They've figured it out. That's not the issue. Scaling up brings about new problems. This article specifically mentions the 45 day rolling restart issue. That's not an issue when…

What exactly do you think requires cycling millions of hosts to take 45 days? That's a couple hundred hosts every five minutes across a large number of datacenters?

I wouldn't expect it to be halved by optimization. I'd expect it to be an order of magnitude faster and take more like 4.5 days.

I wouldn't be surprised (but I would be impressed) if they tried and got it down to a full cycle requiring one working day. That's around 1% of hosts cycling every five minutes.

Re: How Meta patches Linux at hyperscale

#62

Post by hellozomo »

extr wrote: post #17
hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

> My guess would be that this approach is papering over organizational dysfunction. So what? At large enough scale organization problems are harder than technical ones: if you can fix the former with the latter, that's still a win.

But it's not "fixing" the problem, it's papering over the problem by piling tech debt on top of tech debt.

Framing this like it's a good thing is my only objection.

Re: How Meta patches Linux at hyperscale

#63

Post by imetatroll »

faitswulff wrote: post #22

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

The danger in your comment is that people who are primarily "working" because they want to play on someone else's dollar with some new, frivolous tech - from the perspective of what their business actually needs - are going to be insulted by this. Good luck.

Re: How Meta patches Linux at hyperscale

#64

Post by lokar »

hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

At google we did pretty much the same thing. Aimed to be able roll a kernel in 30 days, but various edge cases always made it drag out at the end unless you really spend a lot of human time on it. So use kaplice for really critical stuff (where the patch was easy, not always the case).

A reasonable compromise in the real world.

Re: How Meta patches Linux at hyperscale

#65

Post by ungamedplayer »

lokar wrote: post #64
hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

At google we did pretty much the same thing. Aimed to be able roll a kernel in 30 days, but various edge cases always made it drag out at the end unless you really spend a lot of human time on it. So use kaplice for really critical stuff (where the patch was easy, not always the case). A reasonable compromise in the real world.

At redhat, we just maintain kpatch and hire kernel engineering to do the same. Turn around is about a week for 40 variants of kernel.

Re: How Meta patches Linux at hyperscale

#66

Post by cortesoft »

TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server.

I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because some hardware is going to fail every day just from probability. You have to design everything for failure.

Given that, it shouldn't be hard to take servers out of production for patching and updates.

In other words, a hyperscaler is going to have less incentive to minimize down time than smaller shops.

Re: How Meta patches Linux at hyperscale

#67

Post by vbezhenar »

cortesoft wrote: post #66
TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

I don't follow. Reboot is downtime. Of course your architecture must allow for downtime if it happens, but it's lost money either way, your hardware is not doing any useful work while rebooting. So more computers you have, more money is lost. At small scale that's not significant, but at large scale that might become significant so there's more incentive to reduce downtime.

Re: How Meta patches Linux at hyperscale

#69

Post by jonhohle »

vbezhenar wrote: post #67
cortesoft wrote: post #66

Earlier quoted context omitted.

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

I don't follow. Reboot is downtime. Of course your architecture must allow for downtime if it happens, but it's lost money either way, your hardware is not doing any useful work while rebooting. So more computers you have, more money is lost. At small scale that's not significant, but at large scale that might become significant so there's more incentive to reduce downtime.

A reboot, a software deployment (kernel upgrade), server replacement, etc. are all the same process. That simplifies things dramatically. You can micro-optimize the 30s it takes to reboot a server, or you can simplify a runbook to have one process for any “deployment”. Different scenarios require different things but for most “web scale” things that need to be overprovisioned anyway, I’d take the simpler process.

Re: How Meta patches Linux at hyperscale

#70

Post by tentacleuno »

vbezhenar wrote: post #67
cortesoft wrote: post #66

Earlier quoted context omitted.

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

I don't follow. Reboot is downtime. Of course your architecture must allow for downtime if it happens, but it's lost money either way, your hardware is not doing any useful work while rebooting. So more computers you have, more money is lost. At small scale that's not significant, but at large scale that might become significant so there's more incentive to reduce downtime.

Isn't it more significant at smaller scale? That is, if you have less computers running to serve requests, the downtime of the singular system will be more pronounced (as opposed to rebooting one machine out of 20 in a rack).
Post reply on HN