Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The death of thread per core

buttondown.com

Post reply on HN
Print view View original

61–70 of 70 posts

Re: The death of thread per core

#61

Post by jgraettinger1 »

jandrewrogers wrote: post #39
FridgeSeal wrote: post #14

Earlier quoted context omitted.

Are there any resources/learning material about the more modern thread-per-core approaches? It’s a particular area of interest for me, but I’ve had relatively little success finding more learning material, so I assume there’s lots of tightly guarded institutional knowledge.

Unfortunately, not really. I worked in HPC when it was developed as a concept there, which is where I learned it. I brought it over into databases which was my primary area of expertise because I saw the obvious cross-over application to some scaling challenges in databases. Over time, other people have adopted the ideas but a lot of database R&D is never published. Writing a series of articles about the history and…

As someone with workloads that can benefit from these techniques, but limited resources to put them into practice, my working thesis has been:

* Use a multi-threaded tokio runtime that's allocated a thread-per-core * Focus on application development, so that tasks are well scoped / skewed and don't _need_ stealing in the typical case * Over time, the smart people working on Tokio will apply research to minimize the cost of work-stealing that's not actually needed. * At the limit, where long-lived tasks can be distributed across cores and all cores are busy, the performance will be near-optimal as compared with a true thread-per-core model.

What's your hot take? Are there fundamental optimizations to a modern thread-per-core architecture which seem _impossible_ to capture in a work-stealing architecture like Tokio's?

Re: The death of thread per core

#62

Post by bsenftner »

reactordev wrote: post #38
Keyframe wrote: post #31

Earlier quoted context omitted.

You can't look at it like _that_. Biometrics has its own "things". I don't know what OP is actually doing, but it's probably not classical image processing. Most probably facial features are going through some "form of LGBPHS binarized and encoded which is then fed into an adaptive bloom filter based transform"[0]. Paper quotes 76,800 bits per template (less compressed) and with 64-bit words it's what, 1200 64-bit bi…

Correct, it’s probably distance of a vector or something like that after the bloom. Take the facial points as a vec as you only have a little over a dozen and it’s going to fit nicely in L1.

NDA prevents me from saying anything beyond the compares are minimal representatives of a face template, and those stream through the core's caches.

Re: The death of thread per core

#63

Post by reactordev »

bsenftner wrote: post #62
reactordev wrote: post #38

Earlier quoted context omitted.

Correct, it’s probably distance of a vector or something like that after the bloom. Take the facial points as a vec as you only have a little over a dozen and it’s going to fit nicely in L1.

NDA prevents me from saying anything beyond the compares are minimal representatives of a face template, and those stream through the core's caches.

Queue the “If I were to build it…” ;)

Re: The death of thread per core

#64

Post by menaerus »

AaronAPU wrote: post #13
vacuity wrote: post #2

There are no hard rules; use principles flexibly. That being said, there are some things that are generally true for the long term: use a pinned thread per core, maximize locality (of data and code, wherever relevant), use asynchronous programming if performance is necessary. To incorporate the OP, give control where it's due to each entity (here, the scheduler). Cross-core data movement was never the enemy, but unpr…

I did mass scale performance benchmarking on highly optimized workloads using lockfree queues and fibers, and locking to a core almost never was faster. There were a few topologies where it was, but they were outliers. This was on a wide variety of intel, AMD, NUMA, ARM processors with different architectures, OSes and memory configurations. Part of the reason is hyper threading (or threadripper type archs) but even…

What type of workloads?

Re: The death of thread per core

#65

Post by packetlost »

jandrewrogers wrote: post #39
FridgeSeal wrote: post #14

Earlier quoted context omitted.

Are there any resources/learning material about the more modern thread-per-core approaches? It’s a particular area of interest for me, but I’ve had relatively little success finding more learning material, so I assume there’s lots of tightly guarded institutional knowledge.

Unfortunately, not really. I worked in HPC when it was developed as a concept there, which is where I learned it. I brought it over into databases which was my primary area of expertise because I saw the obvious cross-over application to some scaling challenges in databases. Over time, other people have adopted the ideas but a lot of database R&D is never published. Writing a series of articles about the history and…

I'll toss $20-50 your way to bump up the priority on writing that knowledge down, only strings are it has to actually get done and be publicly available

Re: The death of thread per core

#66

Post by xmodem »

foota wrote: post #21

An interesting observation: "At that time, ensuring maximum CPU utilization was not so important, since you’d typically be bound by other things, but things like disk speed has improved dramatically in the last 10 years while CPU speeds have not."

I'm going to quibble with that observation. CPUs HAVE improved dramatically in the past 10 years. It just doesn't look dramatic if your comparison point is storage speed.

Re: The death of thread per core

#67

Post by bsenftner »

bsenftner wrote: post #62
reactordev wrote: post #38

Earlier quoted context omitted.

Correct, it’s probably distance of a vector or something like that after the bloom. Take the facial points as a vec as you only have a little over a dozen and it’s going to fit nicely in L1.

NDA prevents me from saying anything beyond the compares are minimal representatives of a face template, and those stream through the core's caches.

A public report from the employer about the tech https://cyberextruder.com/wp-content/uploads/2022/06/Accurac... (I no longer work there.)

Re: The death of thread per core

#68

Post by jandrewrogers »

jgraettinger1 wrote: post #61
jandrewrogers wrote: post #39

Earlier quoted context omitted.

Unfortunately, not really. I worked in HPC when it was developed as a concept there, which is where I learned it. I brought it over into databases which was my primary area of expertise because I saw the obvious cross-over application to some scaling challenges in databases. Over time, other people have adopted the ideas but a lot of database R&D is never published. Writing a series of articles about the history and…

As someone with workloads that can benefit from these techniques, but limited resources to put them into practice, my working thesis has been: * Use a multi-threaded tokio runtime that's allocated a thread-per-core * Focus on application development, so that tasks are well scoped / skewed and don't _need_ stealing in the typical case * Over time, the smart people working on Tokio will apply research to minimize the c…

A core assumption underlying thread-per-core architecture is that you will be designing a custom I/O and execution scheduler that is purpose-built for your software and workload at a very granular level. Most expectations of large performance benefits follow from this assumption.

At some point, people started using thread-per-core style while delegating scheduling to a third-party runtime, which almost completely defeats the purpose. If you let tokio et al do that for you, you are leaving a lot of performance and scale on the table. This is an NP-Hard problem; the point of solving it at compile-time is that it is computationally intractable for generic code to create a good schedule at runtime unless it is a trivial case. We need schedulers to consistently make excellent decisions extremely efficiently. I think this point is often lost in discussions of thread-per-core. In the old days we didn’t have runtimes, it was just assumed you would be designing an exotic scheduler. The lack of discussion around this may have led people to believe it wasn’t a critical aspect.

The reality that designing excellent workload-optimized I/O and execution schedulers is an esoteric, high-skill endeavor. It requires enormous amounts of patience and craft, it doesn’t lend itself to quick-and-dirty prototypes. If you aren’t willing to spend months designing the many touch points for the scheduler throughout your software, the algorithms for how events across those touch points interact, and analyzing the scheduler at a systems level for equilibria and boundary conditions then thread-per-core might not be worth the effort.

That said, it isn’t rocket science to design a reasonable schedule for software that is e.g. just taking data off the wire and doing something with it. Most systems are not nearly as complex as e.g. a full-featured database kernel.

Re: The death of thread per core

#69

Post by skavi »

vlovich123 wrote: post #37
pron wrote: post #34

Earlier quoted context omitted.

An occasional successful CAS (on an owned cache line) has very little cost, but if you have to sprinkle atomics/mutexes all over the place , then there's something that's clearly not scalable in your design regardless of the concurrency implementation (you're expecting contention in a lot of places).

An atomic add on a 6ghz high end desktop CPU (13900) is I believe on the order of 4-10ns. If it’s in your hot path your hot path can’t go faster than 50-100 million operations/s - that’s the cost of 1 such instruction in your hotpath (down from the 24 billion non-atomic additions your 6ghz could do otherwise). A CAS brings this down to ~20-50 Mops/s. So it’s quite a meaningful slowdown if you actually want to use the…

Is that best case latency? e.g., with only one thread adding to that location?

Re: The death of thread per core

#70

Post by sumtechguy »

EraYaN wrote: post #60
sumtechguy wrote: post #58

Earlier quoted context omitted.

That is application of the formula Pre-work time + pack up time + send time + unpack time + work time + pack up time + send time + unpack time + post-work time. All remote work has these properties. Even something 'simple' like a remote REST call. If 'remote work time' plus all that other stuff is less than your local calls then it is time wise worth sending it remote. If not local CPU would win. That in many cases r…

There are some neat tricks to remove almost all the pack and unpack time. Apache Arrow can help a ton there (uses the same data format on both CPU and GPU or other accelerator). And on some unified memory systems even the send time can be very low.

oh absolutely reducing any of those spots can change the dynamics of the formula. 'zero copy memory' is another local item many drivers can use. It is just one of those things that also change as tech marches on. What used to be the gold standard on speed is suddenly rendered moot because one of those variables changes. Or suddenly something new will become possible. It is kinda cool but you need to keep an eye on it.
Post reply on HN