How Meta patches Linux at hyperscale • Hacker News Board

Re: How Meta patches Linux at hyperscale

#111

Post by mypalmike » Tue, Dec 05, 2023, 9:06 PM UTC

xyzzy_plugh wrote: post #109

sangnoir wrote: post #107

Earlier quoted context omitted.

> I appreciate your attempt to invalidate my experience but your points are irrelevant That was not my intention - I genuinely would have appreciated answers to my questions as it be useful to compare the complexity of your setup versus Facebook. As an extreme case: million homogenous, stateless hosts are far less complex to manage compared a million heterogeneous, stateful ones, and very little translates from the f…

These are fair points but I'd add that continually draining amortizes to $0 as the fleet grows. Even if you can splice there are benefits to limiting uptime, with maintenance reaping the majority.

The amount of time needed to restart a fleet (without sacrificing availability) is correlated with excess server capacity. Excess server capacity is not free.

Re: How Meta patches Linux at hyperscale

#112

Post by bradknowles » Thu, Dec 07, 2023, 3:14 AM UTC

cortesoft wrote: post #66

TheAceOfHearts wrote: post #27

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by ju…

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

If you've seen any of Werner Vogel's talks, you will notice that Amazon and AWS feel the same way. Any scalable service should be able to withstand the loss of some of their components and keep operating.

Re: How Meta patches Linux at hyperscale

#113

Post by bradknowles » Thu, Dec 07, 2023, 3:15 AM UTC

nickstinemates wrote: post #74

jonhohle wrote: post #69

Earlier quoted context omitted.

A reboot, a software deployment (kernel upgrade), server replacement, etc. are all the same process. That simplifies things dramatically. You can micro-optimize the 30s it takes to reboot a server, or you can simplify a runbook to have one process for any “deployment”. Different scenarios require different things but for most “web scale” things that need to be overprovisioned anyway, I’d take the simpler process.

These servers don't take 30s to reboot. Some servers take many minutes. It's a lot.

Spin new servers up before you take the old ones down. Effectively zero loss of time for that service.

Re: How Meta patches Linux at hyperscale

#114

Post by bradknowles » Thu, Dec 07, 2023, 3:17 AM UTC

solatic wrote: post #98

cortesoft wrote: post #66

Earlier quoted context omitted.

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

> You have to design everything for failure. Given that, it shouldn't be hard to take servers out of production for patching and updates. There's a big, big difference, especially at the scale of hundreds of thousands / millions of servers, between designing such that your architecture can suffer 1% of servers being offline and 10% of servers being offline. If you have 1 million servers, even if you could take 1% of…

I think AWS does pretty well with that philosophy at hyper scale.

AWS has millions of servers in a single AZ.