It's worth mentioning that sched_ext was developed at Meta. The schedulers are developed by several companies who collaborate to develop them, not just Meta or Valve or Italia and the development is done in a shared GitHub repo - https://github.com/sched-ext/scx.
Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
51–60 of 407 posts
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#52Earlier quoted context omitted.
I was at Microsoft during the Windows 8 cycle. I remember hearing about a kernel feature I found interesting. Then I found linux had it for a few years at the time. I think the reality is that Linux is ahead on a lot of kernel stuff. More experimentation is happening.
yeah, but you have IO Completion Ports… IO_Uring is still a pale imitation :(
io_uring does more than IOCP. It's more like an asynchronous syscall interface that avoids the overhead of directly trapping into the kernel. This avoids some overheads IOCP cannot. I'm rusty on the details but the NT kernel has since introduced an imitation: https://learn.microsoft.com/en-us/windows/win32/api/ioringap...
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#53Earlier quoted context omitted.
I mean, part of it is that Linux's default scheduler is braindead by modern standards: https://en.wikipedia.org/wiki/Completely_Fair_Scheduler
Part of that is the assumption that Amazon/Meta/Google all have dedicated engineers who should be doing nothing but tuning performance for 0.0001% efficiency gains. At the scale of millions of servers, those tweaks add up to real dollar savings, and I suspect little of how they run is stock.
This is really just an example of survivorship bias and the power of Valve's good brand value. Big tech does in fact employ plenty of people working on the kernel to make 0.1% efficiency gains (for the reason you state), it's just not posted on HN. Someone would have found this eventually if not Valve.
And the people at FB who worked to integrate Valve's work into the backend and test it and measure the gains are the same people who go looking for these kernel perf improvements all day.
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#54Earlier quoted context omitted.
I was at Microsoft during the Windows 8 cycle. I remember hearing about a kernel feature I found interesting. Then I found linux had it for a few years at the time. I think the reality is that Linux is ahead on a lot of kernel stuff. More experimentation is happening.
yeah, but you have IO Completion Ports… IO_Uring is still a pale imitation :(
IOCP is great and was ahead of Linux for decades, but io_uring is also great. It's a different model, not a poor copy.
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#55Valve is practically singlehandedly dragging the Linux ecosystem forward in areas that nobody else wanted to touch. They needed Windows games to run on Linux so we got massive Proton/Wine advancements. They needed better display output for the deck and we got HDR and VRR support in wayland. They also needed smoother frame pacing and we got a scheduler that Zuck is now using to run data centers. Its funny to think tha…
My favourite is the Windows futex primitives being shipped on Linux: https://lwn.net/Articles/961884/
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#56Earlier quoted context omitted.
Over time they're going to touch things that people were waiting for Microsoft to do for years. I don't have an example in mind at the moment, but it's a lot better to make the changes yourself than wait for OS or console manufacturer to take action.
I was at Microsoft during the Windows 8 cycle. I remember hearing about a kernel feature I found interesting. Then I found linux had it for a few years at the time. I think the reality is that Linux is ahead on a lot of kernel stuff. More experimentation is happening.
And behind on a lot of stuff. The Microsoft's ACLs are nothing short of one of the best designed permission systems there are.
On the surface, they are as simple as Linux UOG/rwx stuff if you want it to be, but you can really, REALLY dive into the technology and apply super specific permissions.
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#57Interesting to see server workloads take ideas from other areas. I saw recently that some of the k8s specific os do their updates like android devices
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#58I'm struggling to understand what workloads Meta might be running that are _this_ latency-critical.
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#59Valve is practically singlehandedly dragging the Linux ecosystem forward in areas that nobody else wanted to touch. They needed Windows games to run on Linux so we got massive Proton/Wine advancements. They needed better display output for the deck and we got HDR and VRR support in wayland. They also needed smoother frame pacing and we got a scheduler that Zuck is now using to run data centers. Its funny to think tha…
Over time they're going to touch things that people were waiting for Microsoft to do for years. I don't have an example in mind at the moment, but it's a lot better to make the changes yourself than wait for OS or console manufacturer to take action.
Surely a gaming handheld counts
Re: Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
#60Earlier quoted context omitted.
This isn’t explicitly called out in any of the other comments in my opinion so I’ll state this. Valve as a company is incredibly focused internally on its business. Its business is games, game hardware, and game delivery. For anything outside of that purview instead of trying to build a huge internal team they contract out. I’m genuinely curious why other companies don’t do this style more often because it seems incr…
I feel like I rarely see contacting out work go well. This seems like an exception
If you have competent people on both sides who care, I don't see why it wouldn't work.
The problem seems, at least from a distance, to be that bosses treat it as a fire-and-forget solution.
We haven't had any software done by oursiders yet, but we have hired consultants to help us on specifics, like changing our infra and help move local servers to the cloud. They've been very effective and helped us a lot.
We had talks though so we found someone who we could trust had the knowledge, and we were knowledgeable enough ourselves that we could determine that. We then followed up closely.