Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Linux Scheduler: A Decade of Wasted Cores (2016)

blog.acolyer.org

Post reply on HN
Print view View original

31–36 of 36 posts

Re: The Linux Scheduler: A Decade of Wasted Cores (2016)

#31

Post by thechao »

brendangregg wrote: post #28
bogomipz wrote: post #27

Earlier quoted context omitted.

>"I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this." Then why make a cryptic comment like you did? Cryptic comments kind of invite questions.

I've already answered it elsewhere -- more than once -- and I'm trying not to waste any more time on it. I think the Linux Foundation is best positioned to coordinate a detailed written response. There's some extra research I did that I haven't shared yet, including, for example, when the bugs were introduced (still needs double checking): - Bug 1: Mar 2011, for Linux 2.6.38 - Bug 2: Apr 2012, for Linux 3.4 - Bug 3:…

Ah, but they were very intense years, so they counted more than one!

Re: The Linux Scheduler: A Decade of Wasted Cores (2016)

#32

Post by velox_io »

lobster_johnson wrote: post #29
awalton wrote: post #21

Earlier quoted context omitted.

The tl;dr is that unless you had an HPC workload with a NUMA box, you couldn't observe this bad behavior. Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.

Is NUMA that rare? Back in 2007-2008 or so, my company bought some rack servers fitted with 48-core AMD Opteron (Magny-Cours), which weren't particularly expensive, and had a NUMA architecture. We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.

You may find this a fascinating read. It's regarding how SQL Server has it's own threading scheduler to bypass the Windows scheduler. When ran on a machine with HyperThreading enabled it will only use 50% of the CPU, I guess they're going after CPU Cache utilisation.

It's really hard to find a decent article on the subject, here's two of the best I've found. https://www.mssqltips.com/sqlservertip/4403/understanding-sq... https://docs.microsoft.com/en-us/sql/relational-databases/th...

Re: The Linux Scheduler: A Decade of Wasted Cores (2016)

#33

Post by Filligree »

hesdeadjim wrote: post #14
Filligree wrote: post #11

Somewhat relevant: Threadripper CPUs, which are aimed at the high-end consumer market, are NUMA with two memory domains. This makes the overall scheduling problem much harder, to the point that they were built with a special "Disable half the cores" mode and supporting hardware to give both memory banks same-speed access to the remaining ones.

Honestly, I believe the only reason they did this is so that when review sites run benchmarks the Threadripper won't look abnormally slow compared to the other chips out there. I own one and in both work and play I have had zero issues. If I drop a frame here and there in a game due to some memory latency? Eh, could care less. If you can afford a Threadripper you can afford a 1080 Ti and a Gsync monitor to smooth out…

You also can probably afford enough memory that the kernel can schedule your game entirely on one half of the CPU, but I don't know if that sort of scheduling (and defragmentation) is commonly used yet.

Re: The Linux Scheduler: A Decade of Wasted Cores (2016)

#34

Post by lobster_johnson »

anarazel wrote: post #30
lobster_johnson wrote: post #29

Earlier quoted context omitted.

Is NUMA that rare? Back in 2007-2008 or so, my company bought some rack servers fitted with 48-core AMD Opteron (Magny-Cours), which weren't particularly expensive, and had a NUMA architecture. We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.

> We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result. I'd bet, but not too much, that that was more due to a) postgres' internal locking implementation scaling horribly at that time b) zone_reclaim_mode leading to bad behaviour around IO.

It's possible, but we weren't testing 48 clients at that time, just our normal workload, which had much less parallelism than that. The person in charge of setting up the systems explained the performance issues as being due to NUMA.

Re: The Linux Scheduler: A Decade of Wasted Cores (2016)

#35

Post by hesdeadjim »

Filligree wrote: post #33
hesdeadjim wrote: post #14

Earlier quoted context omitted.

Honestly, I believe the only reason they did this is so that when review sites run benchmarks the Threadripper won't look abnormally slow compared to the other chips out there. I own one and in both work and play I have had zero issues. If I drop a frame here and there in a game due to some memory latency? Eh, could care less. If you can afford a Threadripper you can afford a 1080 Ti and a Gsync monitor to smooth out…

You also can probably afford enough memory that the kernel can schedule your game entirely on one half of the CPU, but I don't know if that sort of scheduling (and defragmentation) is commonly used yet.

It’s going to be on the application to be NUMA aware, regardless of how much memory you have. Games have never really had to deal with this due to the absolutely minuscule number of people who played games on server-grade dual socket Xeons. It’ll be interesting to see if any of the big names (Unity, Unreal, Crytek/Lunberyard) ever care enough to make a patch for proper NUMA support.

Re: The Linux Scheduler: A Decade of Wasted Cores (2016)

#36

Post by Filligree »

hesdeadjim wrote: post #35
Filligree wrote: post #33

Earlier quoted context omitted.

You also can probably afford enough memory that the kernel can schedule your game entirely on one half of the CPU, but I don't know if that sort of scheduling (and defragmentation) is commonly used yet.

It’s going to be on the application to be NUMA aware, regardless of how much memory you have. Games have never really had to deal with this due to the absolutely minuscule number of people who played games on server-grade dual socket Xeons. It’ll be interesting to see if any of the big names (Unity, Unreal, Crytek/Lunberyard) ever care enough to make a patch for proper NUMA support.

It's entirely possible to do this at the OS level. It makes the scheduling problem much harder, yes, but a user can—for example—force their game to run only in one domain using CPU affinity, then somehow trigger the kernel to migrate all its memory to that domain. I know how to do the former, I haven't tried the latter.

It would be more difficult to do it automatically, but if NUMA systems become more common then I see no reason why it shouldn't be tried.

Post reply on HN