Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Meta patches Linux at hyperscale

thenewstack.io

Post reply on HN
Print view View original

81–90 of 114 posts

Re: How Meta patches Linux at hyperscale

#81

Post by latchkey »

nickstinemates wrote: post #74
jonhohle wrote: post #69

Earlier quoted context omitted.

A reboot, a software deployment (kernel upgrade), server replacement, etc. are all the same process. That simplifies things dramatically. You can micro-optimize the 30s it takes to reboot a server, or you can simplify a runbook to have one process for any “deployment”. Different scenarios require different things but for most “web scale” things that need to be overprovisioned anyway, I’d take the simpler process.

These servers don't take 30s to reboot. Some servers take many minutes. It's a lot.

Worse, some just don't come back without manual intervention. Power supplies don't last forever and might run fine while the machine is on, but after a reboot... boom, gone.

Re: How Meta patches Linux at hyperscale

#82

Post by jb_gericke »

hellozomo wrote: post #10

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work.…

Yeah, especially with containerisation and orchestration / Kubernetes, I get that perhaps not everything is viable to containerise, but in 2023 this feels archaic and like a lot of (potentially unnecessary) engineering work.

Re: How Meta patches Linux at hyperscale

#83

Post by latchkey »

fragmede wrote: post #80
Sarkie wrote: post #53

Earlier quoted context omitted.

How was Oracle to work at?

Post acquisition, we moved offices, and used their HR department, and certain product decisions were made for us, but we largely did our own thing, for better or worse, so I don't know how much I can speak about how it was to work at Oracle. We were under Wim Coekaerts who is a big open source guy. Oracle's reputation as a cutthroat legal entity is well deserved, but working at that side of the company it felt unfair…

Funny given that they have been selling open source software for ages...

https://docs.oracle.com/cd/B10463_01/web.904/b10320/apjsvsup...

I believe this was circa 2000, but going on longer than that.

Re: How Meta patches Linux at hyperscale

#84

Post by tverbeure »

zug_zug wrote: post #33
kreeben wrote: post #24

Earlier quoted context omitted.

I'm guessing you're mad about the term but not the concept? And that you do agree, that scaling from 1 to 10 is not the same animal as scaling from 10 to 100? So why, then, call both animals "plain and simple scaling"?

> And that you do agree, that scaling from 1 to 10 is not the same animal as scaling from 10 to 100? So why, then, call both animals "plain and simple scaling"? So you think we need 4 different words for scaling 1-10, 10-100, 100-1000....? Cut it out, you know it's just marketing hype. Not everything needs its own word.

It makes sense to have different words for totally different orders of scaling, yes.

Re: How Meta patches Linux at hyperscale

#85

Post by yjftsjthsd-h »

latchkey wrote: post #81
nickstinemates wrote: post #74

Earlier quoted context omitted.

These servers don't take 30s to reboot. Some servers take many minutes. It's a lot.

Worse, some just don't come back without manual intervention. Power supplies don't last forever and might run fine while the machine is on, but after a reboot... boom, gone.

I'd prefer kexec to kpatch, then

Re: How Meta patches Linux at hyperscale

#86

Post by j16sdiz »

vbezhenar wrote: post #67
cortesoft wrote: post #66

Earlier quoted context omitted.

> Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I feel like this should be the opposite... I don't work at Meta scale, but I do work for a CDN with 10s of thousands of servers, and everything we do is based on the idea that some machines will always be going down, because so…

I don't follow. Reboot is downtime. Of course your architecture must allow for downtime if it happens, but it's lost money either way, your hardware is not doing any useful work while rebooting. So more computers you have, more money is lost. At small scale that's not significant, but at large scale that might become significant so there's more incentive to reduce downtime.

Yes, you need to overprovision the server.a little bit.

But you got a much simpler process.

Process ain't free either.

Re: How Meta patches Linux at hyperscale

#87

Post by yjftsjthsd-h »

fragmede wrote: post #80
Sarkie wrote: post #53

Earlier quoted context omitted.

How was Oracle to work at?

Post acquisition, we moved offices, and used their HR department, and certain product decisions were made for us, but we largely did our own thing, for better or worse, so I don't know how much I can speak about how it was to work at Oracle. We were under Wim Coekaerts who is a big open source guy. Oracle's reputation as a cutthroat legal entity is well deserved, but working at that side of the company it felt unfair…

> but working at that side of the company it felt unfair because, Oracle's open source contributions with VirtualBox, the UEK, and others are lost in the grar over MySQL.

They are also remembered for closing OpenSolaris and shaking people down over the VirtualBox extension pack ( https://www.theregister.com/2019/10/04/oracle_virtualbox_mer... ).

Re: How Meta patches Linux at hyperscale

#88

Post by raverbashing »

hellozomo wrote: post #61
mypalmike wrote: post #50

Earlier quoted context omitted.

> fixing their system for taking hosts in and out of service, so that it's extremely robust and reliable would likely pay big dividends in reliability. Facebook hosts can be robustly cycled. Of course. They've been doing this stuff for years. They've figured it out. That's not the issue. Scaling up brings about new problems. This article specifically mentions the 45 day rolling restart issue. That's not an issue when…

What exactly do you think requires cycling millions of hosts to take 45 days? That's a couple hundred hosts every five minutes across a large number of datacenters? I wouldn't expect it to be halved by optimization. I'd expect it to be an order of magnitude faster and take more like 4.5 days. I wouldn't be surprised (but I would be impressed) if they tried and got it down to a full cycle requiring one working day. Th…

1 Million hosts over 45 days = 15 min per host

That's a very realistic/optimistic number (especially as you do want to wait for all services to be running and marked as healthy)

"Oh but you can batch this" sure, but you don't want too much of a big batch that will make your service slow or want to risk shooting yourself in the foot - like rebooting your whole control plane then figuring out it doesn't work like that

(The 45 days is probably an estimate as well, I'm not sure they actually do that server by server)

Re: How Meta patches Linux at hyperscale

#89

Post by rthnbgrredf »

ravenstine wrote: post #4

Never heard of this "hyperscale" concept before. How is this any different from... scaling?

I dislike the term 'hyperscale' as it lacks concrete meaning. We need a metric akin to kilobytes or megabytes for storage. Terms like 'kiloscale' and 'megascale' could better indicate the scaling range of a service. For instance, scaling a service to a thousand instances and back to zero or rolling out patches to thousand instances could be termed 'kiloscale'.

Re: How Meta patches Linux at hyperscale

#90

Post by sangnoir »

xyzzy_plugh wrote: post #73
hellozomo wrote: post #61

Earlier quoted context omitted.

What exactly do you think requires cycling millions of hosts to take 45 days? That's a couple hundred hosts every five minutes across a large number of datacenters? I wouldn't expect it to be halved by optimization. I'd expect it to be an order of magnitude faster and take more like 4.5 days. I wouldn't be surprised (but I would be impressed) if they tried and got it down to a full cycle requiring one working day. Th…

It's certainly weird. I've worked at ~million host scale where uptime never exceeded a week by default (with the odd carve out for problem child software supplied by vendors) I'm betting it's moreso that teaching developers to write software that tolerates draining properly (or is even able to communicate draining) is too difficult for them so they work around it.

> It's certainly weird. I've worked at ~million host scale where uptime never exceeded a week by default

How many individual teams had software running on your hosts? How many those hosts were stateful, and were fragmented across hundreds or thousands of service groups that had their own fault tolerances and unknown (to infra team) warm-up times. Adding complexity (rolling reboots) to already complex systems is almost never a good idea - at some point, there will be an issue caused by hosts rebooted in the wrong order, or too many hosts of a certain type 2-dependency-levels down being simultaneously offline

Post reply on HN