How Meta patches Linux at hyperscale

Ksplice is the original live patching technology that got bought by Oracle and was later extended to user space programs while I worked there. It's a really neat technology that isn't made obsolete by the move to cloud, since you still don't want to have to restart the whole fleet at scale.

Never heard of this "hyperscale" concept before. How is this any different from... scaling?

In general every order of magnitude brings new challenges. Companies running over a million servers have a lot of problems that smaller ones don't. Also they just have more room to amortize R&D.

"Draining and un-draining hosts is hard."

I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly.

The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work. Disaster will always be looming.

By contrast, fixing their system for taking hosts in and out of service, so that it's extremely robust and reliable would likely pay big dividends in reliability.

My guess would be that this approach is papering over organizational dysfunction. One team can patch all the kernels but one team can't make all the hosts support proper cycling in and out of service. And no one cares to fix it because there's no real incentive to do so. Only cool hacks and new projects are properly rewarded.