Thread Pools on the JVM • Hacker News Board

Re: Thread Pools on the JVM

#21

Post by cogman10 » Mon, Jul 19, 2021, 5:04 PM UTC

christkv wrote: post #15

cogman10 wrote: post #2

Loom can't land fast enough! The current issue the JVM has is that all threads have a corresponding operating system thread. That, unfortunately, is really heavy memory wise and on the OS context switcher. Loom allows java to have threads as light weight as a goroutine. It's going to change the way everything works. You might still have a dedicated CPU bound thread pool (the common fork join pool exists and probably…

Are we coming full circle going back a variant of the original Java green threads?

Yes and no.

The new Loom threads will be much lighter weight than the original Java green threads. Further, the entire IO infrastructure of the JVM is being reworked for Loom to make sure the OS doesn't block the VM's thread. What's more, Loom does M:N threading.

Same concept, very different implementation.

Re: Thread Pools on the JVM

#22

Post by Spivak » Mon, Jul 19, 2021, 5:09 PM UTC

cogman10 wrote: post #2

Loom can't land fast enough! The current issue the JVM has is that all threads have a corresponding operating system thread. That, unfortunately, is really heavy memory wise and on the OS context switcher. Loom allows java to have threads as light weight as a goroutine. It's going to change the way everything works. You might still have a dedicated CPU bound thread pool (the common fork join pool exists and probably…

You are ignoring the downside to green threads which is that it’s cooperative. If the thread doesn’t yield control back to the event loop then the real OS thread backing the loop is now stuck.

Which leads to dirty things like inserting sleep 0 at the top of loops and dealing with really unbalanced scheduling of threads don’t hit yields often enough. Plus with loom it might not be obvious that some function is a yield since it’s meant to be transparent so if you grab a lock and yield you make everyone wait until your scheduled again.

Green threads are great! I love them and they’re the only real solutions to really concurrent IO heavy workloads but it’s not a panacea and trades one kind of discipline for another.

Re: Thread Pools on the JVM

#23

Post by dragontamer » Mon, Jul 19, 2021, 5:13 PM UTC

cogman10 wrote: post #11

dragontamer wrote: post #6

Earlier quoted context omitted.

Threadpools are probably better on CPU-bound bound (or CPU-ish bound tasks: like RAM-bound) without any I/O. Coroutines / Goroutines and the like are probably better on I/O bound tasks where the CPU-effort in task-switching is significant. -------- For example: Matrix Multiplication is better with a Threadpool. Handling 1000 simultaneous connections when you get Slashdotted (or "Hacker News hug of death") is better s…

I agree. Coroutines MIGHT be more efficient if what you end up building is a statemachine anyways (as that's what most of those coroutines are doing with the compiler). Otherwise, if it's just pure parallel CPU/memory burning with little state transitions/dependence then a dedicated CPU pool fixed to roughly the number of CPU cores on the box will be the most efficient. Heck, it can often even yield benefits to "pin"…

The one that gives me a headache is thinking about how to oversubscribe a GPU (or worse: 4 GPUs, as in the case of the Summit supercomputer).

Its I/O to send data to and from a GPU, and therefore its an I/O bound task somewhat. But there's also a significant amount of CPU work involved. Ideally, you want to balance CPU-work and GPU-work to maximize the work being done.

Fortunately, CUDA-streams seems like they'd mesh pretty well with coroutines (if enough code were there to support them). But if you're reaching for the "GPU-button", everything is compute-bound (if not, you're "doing it wrong"). So now you have a question of "how much to oversubscribe?"

Then again, that's why you just make the oversubscription-factor a #define and then test a lot to find the right factor.... EDIT: Or maybe you oversubscribe until the GPU / CPU runs out of VRAM / RAM. Oversubscription isn't really an issue with coroutines that are executed inside of a thread-pool: you aren't spending any CPU-time needlessly task-switching.

Re: Thread Pools on the JVM

#24

Post by cogman10 » Mon, Jul 19, 2021, 5:16 PM UTC

jayd16 wrote: post #13

cogman10 wrote: post #7

Earlier quoted context omitted.

Terminology mostly :D I've not looked into the goroutine implementation, so I couldn't tell you how it compares to what I've read loom is doing. Loom is looking to have some extremely compact stacks which means each new "virtual thread" as they are calling them will end up having bytes worth of memory allocated. Another thing coming with loom that go lacks is "structured concurrency". It's the notion that you might h…

>structured concurrency That's good to hear. You see a lot of these Loom discussions talk about implicit and magical asynchronous execution. I was afraid fine grained thread control would be left out. Its super useful if you want to interface with how most GUI frameworks function (ie a Main thread), or important OS threads like a thread with a bound GL context or what have you.

Yeah, while virtual threads are the bread and butter of Loom, they are also adding a lot of QoL things. In particular, the notion of "ScopedVariables" will be a godsend to a lot of concurrent work I do. It's the notion of "I want this bit of context to be carried through from one thread of execution to the next".

Beyond that, one thing the loom authors have suggested is that when you want to limit concurrency the better way to do that is using concurrency constructs like semaphores rather than relying on a fixed pool size.

Re: Thread Pools on the JVM

#25

Post by clhodapp » Mon, Jul 19, 2021, 5:20 PM UTC

_old_dude_ wrote: post #17

ovis wrote: post #14

Earlier quoted context omitted.

What benefits does loom provide vs using something like cats-effect fibres?

You can actually debug the code you write because you get a real stacktrace, not few frames that shows the underlying implementation.

Admittedly, loom will do much better but cats-effect does try its best within the limitations of the current JVM: https://typelevel.org/cats-effect/docs/2.x/guides/tracing

Re: Thread Pools on the JVM

#26

Post by sudhirj » Mon, Jul 19, 2021, 5:22 PM UTC

Spivak wrote: post #22

cogman10 wrote: post #2

Loom can't land fast enough! The current issue the JVM has is that all threads have a corresponding operating system thread. That, unfortunately, is really heavy memory wise and on the OS context switcher. Loom allows java to have threads as light weight as a goroutine. It's going to change the way everything works. You might still have a dedicated CPU bound thread pool (the common fork join pool exists and probably…

You are ignoring the downside to green threads which is that it’s cooperative. If the thread doesn’t yield control back to the event loop then the real OS thread backing the loop is now stuck. Which leads to dirty things like inserting sleep 0 at the top of loops and dealing with really unbalanced scheduling of threads don’t hit yields often enough. Plus with loom it might not be obvious that some function is a yield…

Sleep 0 sounds like quite a hack, Go has the neater https://pkg.go.dev/runtime#Gosched instead, and I assume there will be a Java equivalent as well. And if most stdlib methods and all blocking methods call it, it's going to be pretty difficult to hang a green thread.

Re: Thread Pools on the JVM

#27

Post by cogman10 » Mon, Jul 19, 2021, 5:23 PM UTC

dragontamer wrote: post #23

cogman10 wrote: post #11

Earlier quoted context omitted.

I agree. Coroutines MIGHT be more efficient if what you end up building is a statemachine anyways (as that's what most of those coroutines are doing with the compiler). Otherwise, if it's just pure parallel CPU/memory burning with little state transitions/dependence then a dedicated CPU pool fixed to roughly the number of CPU cores on the box will be the most efficient. Heck, it can often even yield benefits to "pin"…

The one that gives me a headache is thinking about how to oversubscribe a GPU (or worse: 4 GPUs, as in the case of the Summit supercomputer). Its I/O to send data to and from a GPU, and therefore its an I/O bound task somewhat. But there's also a significant amount of CPU work involved. Ideally, you want to balance CPU-work and GPU-work to maximize the work being done. Fortunately, CUDA-streams seems like they'd mesh…

And, TBF, a lot of the IO stuff comes down to specifically talking about what sort of device you are talking to and where.

For a lot of the programming I do (and I'm sure a lot of others on HN) IO is almost all network IO. For that, because it's so slow and everything is working over DMA anyways, coroutines end up working really well.

However, once you start talking about on system resources such as SSDs or the GPU, it gets more tricky. As you rightly point out, the GPU is especially bad because all GPU communication ends up being routed through the CPU. At least for a HD, there's DMA which cuts down on the amount of CPU work that needs to be done to access a bit of data.

Re: Thread Pools on the JVM

#28

Post by brokencode » Mon, Jul 19, 2021, 5:23 PM UTC

Spivak wrote: post #22

cogman10 wrote: post #2

Loom can't land fast enough! The current issue the JVM has is that all threads have a corresponding operating system thread. That, unfortunately, is really heavy memory wise and on the OS context switcher. Loom allows java to have threads as light weight as a goroutine. It's going to change the way everything works. You might still have a dedicated CPU bound thread pool (the common fork join pool exists and probably…

You are ignoring the downside to green threads which is that it’s cooperative. If the thread doesn’t yield control back to the event loop then the real OS thread backing the loop is now stuck. Which leads to dirty things like inserting sleep 0 at the top of loops and dealing with really unbalanced scheduling of threads don’t hit yields often enough. Plus with loom it might not be obvious that some function is a yield…

I was under the impression that Loom was implementing preemptable lightweight threads. Is that not the case?

Re: Thread Pools on the JVM

#29

Post by dragonwriter » Mon, Jul 19, 2021, 5:24 PM UTC

christkv wrote: post #15

cogman10 wrote: post #2

Loom can't land fast enough! The current issue the JVM has is that all threads have a corresponding operating system thread. That, unfortunately, is really heavy memory wise and on the OS context switcher. Loom allows java to have threads as light weight as a goroutine. It's going to change the way everything works. You might still have a dedicated CPU bound thread pool (the common fork join pool exists and probably…

Are we coming full circle going back a variant of the original Java green threads?

> Are we coming full circle going back a variant of the original Java green threads?

There are basically two kinds of green threads:

(1) N:1, where one OS thread hosts all the application threads, and

(2) M:N, where M application threads are hosted on N OS threads.

Original Java (and Ruby, and lots of other systems before every microcomputer was a multicore parallel system) green threads were N:1, which provide concurrency but not parallelism, which is fine when your underlying system can't do real parallelism anyway.)

Wanting to take advantage of multicore systems (at least, in the Ruby case, for underlying native code) drove a transition to native threads (which you could call an N:N threading model, as application and OS threads are identical.)

But this limits the level of concurrency to the level of parallelism, which can be a regression compared to N:1 models for applications where the level of concurrency that is useful is greater than the level of parallelism available.

What lots of newer systems are driving toward, to solve that, are M:N models, which can leverage all available parallelism but also provide a higher degree of concurrency.

Re: Thread Pools on the JVM

#30

Post by neeleshs » Mon, Jul 19, 2021, 5:28 PM UTC

christkv wrote: post #15

cogman10 wrote: post #2

Loom can't land fast enough! The current issue the JVM has is that all threads have a corresponding operating system thread. That, unfortunately, is really heavy memory wise and on the OS context switcher. Loom allows java to have threads as light weight as a goroutine. It's going to change the way everything works. You might still have a dedicated CPU bound thread pool (the common fork join pool exists and probably…

Are we coming full circle going back a variant of the original Java green threads?

This gives some color - https://blogs.oracle.com/javamagazine/going-inside-javas-pro...