Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The death of thread per core

buttondown.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: The death of thread per core

#51

Post by FridgeSeal »

jandrewrogers wrote: post #39
FridgeSeal wrote: post #14

Earlier quoted context omitted.

Are there any resources/learning material about the more modern thread-per-core approaches? It’s a particular area of interest for me, but I’ve had relatively little success finding more learning material, so I assume there’s lots of tightly guarded institutional knowledge.

Unfortunately, not really. I worked in HPC when it was developed as a concept there, which is where I learned it. I brought it over into databases which was my primary area of expertise because I saw the obvious cross-over application to some scaling challenges in databases. Over time, other people have adopted the ideas but a lot of database R&D is never published. Writing a series of articles about the history and…

> Writing a series of articles about the history and theory of thread-per-core software architecture has been on my eternal TODO list

Your past has already been super interesting, so if you ever do get around to writing this, I’d be very excited to read it!

Re: The death of thread per core

#52

Post by woooooo »

pjmlp wrote: post #42
dist-epoch wrote: post #12

Earlier quoted context omitted.

The thing with GPUs is that for many problems really dumb and simple algorithms (think bubble sort equivalent) are many times faster than very fancy CPU algorithms (think quick sort equivalent). Your typical non-neural-network GPU algorithm is rarely using more than 50% of it's power, yet still outperforms carefully written CPU algorithms.

Except it is only worth doing, if when taking into account loading data into the GPU and getting the results back, is still faster than total execution on the CPU. It doesn't help that GPU beats the CPU in compute, if a plain SIMD approach outperforms the total execution time.

Especially if you're saving watts in the process. And not utilizing a capital-intensive asset.

Re: The death of thread per core

#53

Post by whizzter »

bob1029 wrote: post #47
zipy124 wrote: post #23

Earlier quoted context omitted.

How are you doing tens of millions of faces per second per core, first of all assuming a 5ghz processor, that gives you 500 cycles per image if you do ten million a second, that's not nearly enough to do anything image related. Second of all L1 cache is at most in the hundreds of kilobytes, so the faces aren't in L1 but must be retrieved from elsewhere...??

> assuming a 5ghz processor, that gives you 500 cycles per image if you do ten million a second Modern CPUs don't quite work this way. Many instructions can be retired per clock cycle. > Second of all L1 cache is at most in the hundreds of kilobytes, so the faces aren't in L1 but must be retrieved from elsewhere...?? Yea, from L2 cache. It's caches all the way down. That's how we make it go really fast. The prefetche…

The keyword is CAN, there can also be huge penalties (random main-memory accesses are over a cycles typically), the parent was probably considering a regular image transform/comparison and 20 pixels per cycle even for low resolution 100x100 images is way above what we do today.

As others have mentioned, they're probably doing some kind of embedding like search primarily and then 500 cycles per face makes more sense, but it's not a full comparison.

Re: The death of thread per core

#54

Post by torginus »

jandrewrogers wrote: post #9

I've worked on several thread-per-core systems that were purpose-built for extreme dynamic data and load skew. They work beautifully at very high scales on the largest hardware. The mechanics of how you design thread-per-core systems that provide uniform distribution of load without work-stealing or high-touch thread coordination have idiomatic architectures at this point. People have been putting thread-per-core arc…

If I remember correctly, these work stealing task schedulers started getting pushed around the mid 2000s as a result of Intel failing to scale the Pentium 4 architecture to expected single-thread performance levels.

Libraries like .NET's Task Parallel Library or Intel Threaded Building Blocks pretty much cemented these work-stealing task architectures. It's not that they didn't work well enough, but Intel Core came along, and single-threaded perf scaling was possible again, so these libraries became less of a focus.

It seems multi-core interest is back.

Re: The death of thread per core

#55

Post by codetiger »

bob1029 wrote: post #3

I look at cross core communication as a 100x latency penalty. Everything follows from there. The dependencies in the workload ultimately determine how it should be spread across the cores (or not!). The real elephant in the room is that oftentimes it's much faster to just do the whole job on a single core even if you have 255 others available. Some workloads do not care what kind of clever scheduler you have in hand.…

I was with a similar assumption that thread per core might be the best approach for one of my OpenSource Rust libraries that is a Workflow Orchestration engine. The engine is focused on payment processing. The perv version had thread local engine and focused on thread per core. When I moved to a pure async based engine using tokio runtime and all underlying libraries made thread safe, it improved the performance 2x. The entire workload being fully CPU driven with no IO. I was assuming tokio mostly does better only for IO based workloads, however my tests proved me wrong. Now am not moving away from async approach. https://github.com/GoPlasmatic/dataflow-rs

Re: The death of thread per core

#56

Post by formerly_proven »

pjmlp wrote: post #42
dist-epoch wrote: post #12

Earlier quoted context omitted.

The thing with GPUs is that for many problems really dumb and simple algorithms (think bubble sort equivalent) are many times faster than very fancy CPU algorithms (think quick sort equivalent). Your typical non-neural-network GPU algorithm is rarely using more than 50% of it's power, yet still outperforms carefully written CPU algorithms.

Except it is only worth doing, if when taking into account loading data into the GPU and getting the results back, is still faster than total execution on the CPU. It doesn't help that GPU beats the CPU in compute, if a plain SIMD approach outperforms the total execution time.

The "GPU as accelerator" vs. "GPU-native software" split. The former usually results in or from poor, generic architectures.

Re: The death of thread per core

#57

Post by jcelerier »

bob1029 wrote: post #46
kcb wrote: post #29

Earlier quoted context omitted.

No shot are you doing tens of millions of anything useful per second per core. That's like beyond HFT numbers.

You can handle hundreds of millions of transactions per second if you are thoughtful enough in your engineering. ValueDisruptor in .NET can handle nearly half a billion items per second per core. The Java version is what is typically used to run the actual exchanges (no value types), so we can go even faster if we needed to without moving to some exotic compute or GPU technology.

It's so sad to see how many people not knowing how incredibly fast our CPUs are

Re: The death of thread per core

#58

Post by sumtechguy »

dist-epoch wrote: post #12
bob1029 wrote: post #3

I look at cross core communication as a 100x latency penalty. Everything follows from there. The dependencies in the workload ultimately determine how it should be spread across the cores (or not!). The real elephant in the room is that oftentimes it's much faster to just do the whole job on a single core even if you have 255 others available. Some workloads do not care what kind of clever scheduler you have in hand.…

The thing with GPUs is that for many problems really dumb and simple algorithms (think bubble sort equivalent) are many times faster than very fancy CPU algorithms (think quick sort equivalent). Your typical non-neural-network GPU algorithm is rarely using more than 50% of it's power, yet still outperforms carefully written CPU algorithms.

That is application of the formula

Pre-work time + pack up time + send time + unpack time + work time + pack up time + send time + unpack time + post-work time.

All remote work has these properties. Even something 'simple' like a remote REST call. If 'remote work time' plus all that other stuff is less than your local calls then it is time wise worth sending it remote. If not local CPU would win.

That in many cases right now the GPU is 'winning' that race.

Re: The death of thread per core

#59

Post by justincormack »

FridgeSeal wrote: post #14
jandrewrogers wrote: post #9

I've worked on several thread-per-core systems that were purpose-built for extreme dynamic data and load skew. They work beautifully at very high scales on the largest hardware. The mechanics of how you design thread-per-core systems that provide uniform distribution of load without work-stealing or high-touch thread coordination have idiomatic architectures at this point. People have been putting thread-per-core arc…

Are there any resources/learning material about the more modern thread-per-core approaches? It’s a particular area of interest for me, but I’ve had relatively little success finding more learning material, so I assume there’s lots of tightly guarded institutional knowledge.

The ScyllaDB team wrote a bunch (and p99conf is on today)

Re: The death of thread per core

#60

Post by EraYaN »

sumtechguy wrote: post #58
dist-epoch wrote: post #12

Earlier quoted context omitted.

The thing with GPUs is that for many problems really dumb and simple algorithms (think bubble sort equivalent) are many times faster than very fancy CPU algorithms (think quick sort equivalent). Your typical non-neural-network GPU algorithm is rarely using more than 50% of it's power, yet still outperforms carefully written CPU algorithms.

That is application of the formula Pre-work time + pack up time + send time + unpack time + work time + pack up time + send time + unpack time + post-work time. All remote work has these properties. Even something 'simple' like a remote REST call. If 'remote work time' plus all that other stuff is less than your local calls then it is time wise worth sending it remote. If not local CPU would win. That in many cases r…

There are some neat tricks to remove almost all the pack and unpack time. Apache Arrow can help a ton there (uses the same data format on both CPU and GPU or other accelerator). And on some unified memory systems even the send time can be very low.
Post reply on HN