Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thread Pools on the JVM

gist.github.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: Thread Pools on the JVM

#111

Post by samus »

Nullabillity wrote: post #55
elygre wrote: post #48

Earlier quoted context omitted.

What is broken-by-design about the api?

Fundamentally, an async API is either data-oriented (Futures/Promises: tell me what data this task produced) or job-oriented (Threads: tell me when this task is done). You can think of it like functions vs subroutines. Since you typically care about the data produced by the task, threads require you to sort out your own backchannel for communicating this data back (such as: a channel, a mutexed variable, or something…

Project Loom's scope explicitly encompasses not only virtual threads. To do that, the concept of structured concurrency[1] was introduced. There /are/ going to be new APIs.

[1]: http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2.h...

Re: Thread Pools on the JVM

#112

Post by cbsmith »

samus wrote: post #110
kllrnohj wrote: post #69

Earlier quoted context omitted.

> So, there was a time where a broad statement like that was pretty solid. That time is approaching 20 years old at this point, too. Native threads haven't been "expensive" for a very, very long time now. Maybe if you're in the camp of disabling overcommit it matters, but otherwise the application of green threads is definitely a specialized niche, not generally useful. > In the broad set of use cases though, switchi…

> I'd go even further and say it'll be a net-loss in most cases, especially with modern complications like heterogeneous compute. If you're use case is specifically spinning up thousands of threads for IO (aka, you're a server & nothing else), then sure. But if you aren't there's no win here, just complications (like times when you need native thread isolation for FFI reasons, like using OpenGL) Virtual threads are g…

Having it be optional increases, not decreases, complexity. ;-) It also increases the propensity for people to use the feature blindly.

Re: Thread Pools on the JVM

#113

Post by lmm »

pron wrote: post #106
lmm wrote: post #96

Earlier quoted context omitted.

Isn't the point of preemption to degrade gracefully when the cores are oversubscribed? E.g. the first system I worked on ran potentially CPU-heavy work from various clients, and used per-client threads to isolate them; every so often clients would find ways to get their thread stuck doing a large amount of CPU work (e.g. regex backtracking) and although these were in some sense bugs (and we did fix them), it was very…

There's no doubt forced preemption could help, but I'm still unsure about what the right algorithm is; probably not time sharing. Suppose you have 100K threads, and only 1% of them become CPU-bound for 100ms. That could take down your 32-core server for 3 seconds, which is bad. But suppose we had 10ms time-slices. Then, those busy threads' latency might go from 100ms to as high as a few minutes, which means effective…

Sure. Ultimately you've got the same problem as an OS scheduler and recognising whether threads are CPU-bound or IO-bound and treating them separately is probably going to be part of that.

I appreciate not wanting to do things until you can do them right, but equally if you advertise this as a preemptive runtime, people are going to expect that they can use it to throw 32 CPU-spinning threads onto 8 cores and have it behave gracefully. It sounds like from a user's point of view on day 1 this runtime will be the worst of both worlds - you need to take care to not do big chunks of CPU work without yielding, but you don't get the full control that a traditional "userspace" cooperative multitasking framework would give you.

Re: Thread Pools on the JVM

#114

Post by samus »

cbsmith wrote: post #112
samus wrote: post #110

Earlier quoted context omitted.

> I'd go even further and say it'll be a net-loss in most cases, especially with modern complications like heterogeneous compute. If you're use case is specifically spinning up thousands of threads for IO (aka, you're a server & nothing else), then sure. But if you aren't there's no win here, just complications (like times when you need native thread isolation for FFI reasons, like using OpenGL) Virtual threads are g…

Having it be optional increases, not decreases, complexity. ;-) It also increases the propensity for people to use the feature blindly.

For the JVM developers for sure. Implementing Project Loom must have been quite a ride. But even if it is used blindly, there are only three obvious issues I see:

* It's a no-no for computational workloads. As you said, they are concurrent, but not necessarily parallel.

* As you said, care has to be taken to use the right thread when interacting certain low-level APIs.

* It becomes easier to overload upstream systems by sending too many queries concurrently.

Re: Thread Pools on the JVM

#115

Post by pron »

lmm wrote: post #113
pron wrote: post #106

Earlier quoted context omitted.

There's no doubt forced preemption could help, but I'm still unsure about what the right algorithm is; probably not time sharing. Suppose you have 100K threads, and only 1% of them become CPU-bound for 100ms. That could take down your 32-core server for 3 seconds, which is bad. But suppose we had 10ms time-slices. Then, those busy threads' latency might go from 100ms to as high as a few minutes, which means effective…

Sure. Ultimately you've got the same problem as an OS scheduler and recognising whether threads are CPU-bound or IO-bound and treating them separately is probably going to be part of that. I appreciate not wanting to do things until you can do them right, but equally if you advertise this as a preemptive runtime, people are going to expect that they can use it to throw 32 CPU-spinning threads onto 8 cores and have it…

But the experience people have already been having with the Early Access is overwhelmingly positive. Even without forced preemption, "preemptive" is far less misleading than cooperative, even considering the common confusion between preemptive scheduling and time-sharing.

While OS threads might indeed handle 32 spinning threads on an 8-core machine more gracefully, switching between implementations of threads is easy so such a "mistake" is inconsequential, and no OS handles 320,000 spinning threads gracefully, and people know that that is the scale of threads that virtual threads exist to serve.

Re: Thread Pools on the JVM

#116

Post by lmm »

pron wrote: post #115
lmm wrote: post #113

Earlier quoted context omitted.

Sure. Ultimately you've got the same problem as an OS scheduler and recognising whether threads are CPU-bound or IO-bound and treating them separately is probably going to be part of that. I appreciate not wanting to do things until you can do them right, but equally if you advertise this as a preemptive runtime, people are going to expect that they can use it to throw 32 CPU-spinning threads onto 8 cores and have it…

But the experience people have already been having with the Early Access is overwhelmingly positive. Even without forced preemption, "preemptive" is far less misleading than cooperative, even considering the common confusion between preemptive scheduling and time-sharing. While OS threads might indeed handle 32 spinning threads on an 8-core machine more gracefully, switching between implementations of threads is easy…

You're right that calling it "cooperative" would be worse. Still, I suspect Early Access users are paying a lot more attention to the details (and are more knowledgeable users in general) than GA users will; switching thread implementation might be "easy", but I suspect most users will want to use Loom without tuning anything at all. So safe defaults are very important (and I'd suggest that for the default config, safely handling 32 spinning threads on 8 cores is more important than handling 320,000 mostly-sleeping IO-bound threads).

Which is not to say I have a better idea (other than "make the defaults magically do everything right", which is obviously hard).

Re: Thread Pools on the JVM

#117

Post by pron »

lmm wrote: post #116
pron wrote: post #115

Earlier quoted context omitted.

But the experience people have already been having with the Early Access is overwhelmingly positive. Even without forced preemption, "preemptive" is far less misleading than cooperative, even considering the common confusion between preemptive scheduling and time-sharing. While OS threads might indeed handle 32 spinning threads on an 8-core machine more gracefully, switching between implementations of threads is easy…

You're right that calling it "cooperative" would be worse. Still, I suspect Early Access users are paying a lot more attention to the details (and are more knowledgeable users in general) than GA users will; switching thread implementation might be "easy", but I suspect most users will want to use Loom without tuning anything at all. So safe defaults are very important (and I'd suggest that for the default config, sa…

I think we have the best defaults currently possible for the use-cases Loom targets, say, more than a few thousand concurrent tasks. The cases where you might observe some downside compared to the OS (before we choose to expose forced preemption) are not in that class. The only thing to consider is whether you have many concurrent tasks or a few, and if the answer is many, the choice is simple. Otherwise, you can experiment with different implementations, but the few-tasks case is not our initial focus.

Having said that, I'm interested in hearing about real-world cases (involving many tasks, not 32) where forced preemption, and possibly time sharing or maybe another strategy, can be useful. The "accidentally misbehaving subset" is a good example, but time-sharing probably isn't what we need to address it.

Re: Thread Pools on the JVM

#118

Post by 0xffff2 »

cogman10 wrote: post #4
0xffff2 wrote: post #3

This seems like good advice in general. Is any of it really specific to the JVM? If I was doing thread pooling with CPU and IO bound tasks, I would approach threading in a similar way in C++.

It'll depend on if your language has either coroutines or lightweight threads. Threadpooling only matters if you have neither of those things. Otherwise, you should be using one or the other over a thread pool. You might still spin up a threadpool for CPU bound operations, but you wouldn't have one dedicated to IO. As of C++ 20, there are coroutines which you should be looking at (IMO). https://en.cppreference.com/w/…

>As of C++ 20, there are coroutines which you should be looking at (IMO).

Ha! Maybe in 20 years. Sadly, I'm still writing new code targeting C++98 on one project. The most current project I'm a part of is on C++11.

Re: Thread Pools on the JVM

#119

Post by cbsmith »

samus wrote: post #114
cbsmith wrote: post #112

Earlier quoted context omitted.

Having it be optional increases, not decreases, complexity. ;-) It also increases the propensity for people to use the feature blindly.

For the JVM developers for sure. Implementing Project Loom must have been quite a ride. But even if it is used blindly, there are only three obvious issues I see: * It's a no-no for computational workloads. As you said, they are concurrent, but not necessarily parallel. * As you said, care has to be taken to use the right thread when interacting certain low-level APIs. * It becomes easier to overload upstream systems…

Oh there's a bunch of other problems as well. Developers will "solve" problems by increasing the number of virtual threads, that actually should be solved in other ways. Tons of code is going to suddenly discover assumptions about its underlying runtime model are no longer true, leading to subtle and potentially complex problems. New software will need to either take on the burden of choosing a runtime model or adopt the complexity from having to consider a mixture of both...

Re: Thread Pools on the JVM

#120

Post by iamcreasy »

cratermoon wrote: post #100
iamcreasy wrote: post #43

Earlier quoted context omitted.

So, with Loom now we can tell exactly in which order theses threads were executed as it's not up to OS to decide thread execution order anymore?

> we can tell exactly in which order theses threads were executed Everything I've ever been taught about multi-threading, parallelism, and concurrency says never to make any assumptions about execution order. What are you doing that your care about it?

Replay system.
Post reply on HN