Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta patches Linux at hyperscale

thenewstack.io

Post reply on HN
Print view View original

101–110 of 114 posts

Re: How Meta patches Linux at hyperscale

#101

Post by xyzzy_plugh »

sangnoir wrote: post #90
xyzzy_plugh wrote: post #73

Earlier quoted context omitted.

It's certainly weird. I've worked at ~million host scale where uptime never exceeded a week by default (with the odd carve out for problem child software supplied by vendors) I'm betting it's moreso that teaching developers to write software that tolerates draining properly (or is even able to communicate draining) is too difficult for them so they work around it.

> It's certainly weird. I've worked at ~million host scale where uptime never exceeded a week by default How many individual teams had software running on your hosts? How many those hosts were stateful, and were fragmented across hundreds or thousands of service groups that had their own fault tolerances and unknown (to infra team) warm-up times. Adding complexity (rolling reboots) to already complex systems is almos…

I appreciate your attempt to invalidate my experience but your points are irrelevant.

> hosts rebooted in the wrong order

Order doesn't matter. Host groups set a threshold for unavailability. Hosts are not rolled unless availability targets are maintainable. Usually this just means the oldest host at any time will get rolled.

Facebook obviously uses hot patching kernel updates to work around a social issue. Instead if you are functionally able to prescribe a set of behaviors that teams must comply to, you can easily do things like rebooting the fleet monthly without impacting availability regardless of the statefulness or fault tolerances.

If I shoot a random host in your pool and it matters to you then you haven't achieved fault tolerance. I'm obviously not proposing shooting an unfair number of hosts to you.

Re: How Meta patches Linux at hyperscale

#102

Post by phyzome »

agrajag wrote: post #30
phyzome wrote: post #20

Earlier quoted context omitted.

Yeah, and this isn't a problem that interacts with scale. If you can patch 100 servers with an automated method you can patch 1,000,000 of them.

This is absolutely a problem that interacts with scale. With 1M servers you’re almost certainly dealing with hundreds of service owners, and some of those are going to need additional features you don’t have to worry about with 100 servers. Some examples are databases with graceful failover, long running AI model training jobs, or distributed databases like etc where you have to be mindful about how many can be down…

What you're describing has nothing to do with the number of servers and everything to do with the number of services.

That's not scale, that's organizational sprawl.

Re: How Meta patches Linux at hyperscale

#103

Post by cortesoft »

MarkSweep wrote: post #75
cortesoft wrote: post #66

Earlier quoted context omitted.

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

Edit: ignore the below numbers, I got hours and minutes confused. If you each sever spends 7.5 minutes each month rebooting, that is 1% of all your computers wasted. If you have 10,000 servers, that’s worth 100 severs. If you have 1 million servers, that’s worth 10,000 servers. If each server costs $10,000, that’s $100 million dollars of compute capacity. You can see how that amount of lost computer capacity can star…

Capacity is measured by peak usage, but most data centers experience a daily traffic cycle with low times at night. We just patch/upgrade our servers at the local low time, when traffic is much lower and a lot of your machines are idle.

Re: How Meta patches Linux at hyperscale

#104

Post by cortesoft »

vbezhenar wrote: post #67
cortesoft wrote: post #66

Earlier quoted context omitted.

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

I don't follow. Reboot is downtime. Of course your architecture must allow for downtime if it happens, but it's lost money either way, your hardware is not doing any useful work while rebooting. So more computers you have, more money is lost. At small scale that's not significant, but at large scale that might become significant so there's more incentive to reduce downtime.

If it isn’t an emergency patch, we do all our maintenance at low traffic times (e.g. the middle of the night local time for the data center). Your capacity planning is based on peak traffic, so you can afford to have more machines out during low traffic times.

Re: How Meta patches Linux at hyperscale

#106

Post by fragmede »

adrianN wrote: post #71
fragmede wrote: post #41

Earlier quoted context omitted.

Ads are just what pays the bills. The cluster is also used for all sorts of useful things, like mass communication, especially during emergencies, support groups, and providing information to help manage a pandemic. Software engineers aren't nuclear physicists or machinists. I'm not sure how you want to use them to build nuclear reactors.

People become software engineers instead of choosing a different profession because there is so much money in ads.

And lawyers only choose that profession to ambulance chase, and doctors only choose that profession to do boob jobs. People get into software development for more than just money, y'know?

Re: How Meta patches Linux at hyperscale

#107

Post by sangnoir »

xyzzy_plugh wrote: post #101
sangnoir wrote: post #90

Earlier quoted context omitted.

> It's certainly weird. I've worked at ~million host scale where uptime never exceeded a week by default How many individual teams had software running on your hosts? How many those hosts were stateful, and were fragmented across hundreds or thousands of service groups that had their own fault tolerances and unknown (to infra team) warm-up times. Adding complexity (rolling reboots) to already complex systems is almos…

I appreciate your attempt to invalidate my experience but your points are irrelevant. > hosts rebooted in the wrong order Order doesn't matter. Host groups set a threshold for unavailability. Hosts are not rolled unless availability targets are maintainable. Usually this just means the oldest host at any time will get rolled. Facebook obviously uses hot patching kernel updates to work around a social issue. Instead i…

> I appreciate your attempt to invalidate my experience but your points are irrelevant

That was not my intention - I genuinely would have appreciated answers to my questions as it be useful to compare the complexity of your setup versus Facebook. As an extreme case: million homogenous, stateless hosts are far less complex to manage compared a million heterogeneous, stateful ones, and very little translates from the former to the latter - in my experience.

> Facebook obviously uses hot patching kernel updates to work around a social issue.

Which I think is reasonable when you have tens of thousands of SDEs.

> I'm obviously not proposing shooting an unfair number of hosts to you.

I agree with you, but I'll go on to say "not shooting an unfair number of hosts" is a hard problem to solve at scale, unless you're willing to make it simple and make humans deal with it by continually draining/undraining services which costs a lot of money without increasing the top line, likely far more money than it cost to get a handful of engineers to write kernel splicing. So beyond it being possibly a social issue, it may be a cost/host utilization issue as well

Re: How Meta patches Linux at hyperscale

#108

Post by BlackFingolfin »

raverbashing wrote: post #88
hellozomo wrote: post #61

Earlier quoted context omitted.

What exactly do you think requires cycling millions of hosts to take 45 days? That's a couple hundred hosts every five minutes across a large number of datacenters? I wouldn't expect it to be halved by optimization. I'd expect it to be an order of magnitude faster and take more like 4.5 days. I wouldn't be surprised (but I would be impressed) if they tried and got it down to a full cycle requiring one working day. Th…

1 Million hosts over 45 days = 15 min per host That's a very realistic/optimistic number (especially as you do want to wait for all services to be running and marked as healthy) "Oh but you can batch this" sure, but you don't want too much of a big batch that will make your service slow or want to risk shooting yourself in the foot - like rebooting your whole control plane then figuring out it doesn't work like that…

That's not 15 minutes per host; that's 15 hosts per minute.

Re: How Meta patches Linux at hyperscale

#109

Post by xyzzy_plugh »

sangnoir wrote: post #107
xyzzy_plugh wrote: post #101

Earlier quoted context omitted.

I appreciate your attempt to invalidate my experience but your points are irrelevant. > hosts rebooted in the wrong order Order doesn't matter. Host groups set a threshold for unavailability. Hosts are not rolled unless availability targets are maintainable. Usually this just means the oldest host at any time will get rolled. Facebook obviously uses hot patching kernel updates to work around a social issue. Instead i…

> I appreciate your attempt to invalidate my experience but your points are irrelevant That was not my intention - I genuinely would have appreciated answers to my questions as it be useful to compare the complexity of your setup versus Facebook. As an extreme case: million homogenous, stateless hosts are far less complex to manage compared a million heterogeneous, stateful ones, and very little translates from the f…

These are fair points but I'd add that continually draining amortizes to $0 as the fleet grows.

Even if you can splice there are benefits to limiting uptime, with maintenance reaping the majority.

Re: How Meta patches Linux at hyperscale

#110

Post by jononor »

nickstinemates wrote: post #74
jonhohle wrote: post #69

Earlier quoted context omitted.

A reboot, a software deployment (kernel upgrade), server replacement, etc. are all the same process. That simplifies things dramatically. You can micro-optimize the 30s it takes to reboot a server, or you can simplify a runbook to have one process for any “deployment”. Different scenarios require different things but for most “web scale” things that need to be overprovisioned anyway, I’d take the simpler process.

These servers don't take 30s to reboot. Some servers take many minutes. It's a lot.

Sounds like something to fix rather than to paper over?
Post reply on HN