21–30 of 36 posts
Still 0% performance wins for Netflix. Not a "decade of wasted cores" for us.
Can you please explain that more?
Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.
Earlier quoted context omitted.
Can you please explain that more?
The tl;dr is that unless you had an HPC workload with a NUMA box, you couldn't observe this bad behavior. Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.
The Linpack benchmark is an example of an HPC code that should run one process per core and be pinned.
Earlier quoted context omitted.
From my experience, Linux CPU and I/O scheduling got good from 2.6.x (also around 2007) around when AIO became robust.
Isn't AIO still implemented with threads in libc? In that case, it's just the CPU scheduler that counts...
Still 0% performance wins for Netflix. Not a "decade of wasted cores" for us.
Can you please explain that more?
The better question is: why has the Linux Foundation been silent about it?
If this were true -- "a decade of wasted cores", with losses of "13-24% for typical Linux workloads" (for a decade, as the title suggests) -- then companies like Netflix would have lost many millions due to our choice of Linux, and due to the Linux scheduling maintainers and community failing to identify such egregious problems. The industry as a whole -- including every device and server that runs Linux -- would have lost many BILLIONS. It would be one of the most costly failures in technology EVER.
And the Linux Foundation is silent about this? Seriously?
DBAs knew about this for many years now. We would simply change the schedular on the DB servers.
I think you might be confusing CPU schedulers and IO schedulers. Linux never had switchable CPU schedulers.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
fair.c deadline.c rt.c
Still 0% performance wins for Netflix. Not a "decade of wasted cores" for us.
https://news.ycombinator.com/item?id=11573375 https://news.ycombinator.com/item?id=11502221
Earlier quoted context omitted.
Can you please explain that more?
I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this. The better question is: why has the Linux Foundation been silent about it? If this were true -- "a decade of wasted cores", with losses of "13-24% for typical Linux workloads" (for a decade, as the title suggests) -- then companies like Netflix would have lost many millions due to our choice of Linux…
Then why make a cryptic comment like you did? Cryptic comments kind of invite questions.
Earlier quoted context omitted.
I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this. The better question is: why has the Linux Foundation been silent about it? If this were true -- "a decade of wasted cores", with losses of "13-24% for typical Linux workloads" (for a decade, as the title suggests) -- then companies like Netflix would have lost many millions due to our choice of Linux…
>"I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this." Then why make a cryptic comment like you did? Cryptic comments kind of invite questions.
There's some extra research I did that I haven't shared yet, including, for example, when the bugs were introduced (still needs double checking):
- Bug 1: Mar 2011, for Linux 2.6.38
- Bug 2: Apr 2012, for Linux 3.4
- Bug 3: Dec 2009, for Linux 2.6.32
- Bug 4: Feb 2015, for Linux 3.19
Earlier quoted context omitted.
Can you please explain that more?
The tl;dr is that unless you had an HPC workload with a NUMA box, you couldn't observe this bad behavior. Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.
We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.
Earlier quoted context omitted.
The tl;dr is that unless you had an HPC workload with a NUMA box, you couldn't observe this bad behavior. Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.
Is NUMA that rare? Back in 2007-2008 or so, my company bought some rack servers fitted with 48-core AMD Opteron (Magny-Cours), which weren't particularly expensive, and had a NUMA architecture. We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.
I'd bet, but not too much, that that was more due to a) postgres' internal locking implementation scaling horribly at that time b) zone_reclaim_mode leading to bad behaviour around IO.