Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Java Virtual Threads: A Case Study

infoq.com

Post reply on HN
Print view View original

21–30 of 199 posts

Re: Java Virtual Threads: A Case Study

#21

Post by chipdart »

exabrial wrote: post #2

What is the virtual thread / event loop pattern seeking to optimize? Is it context switching? A number of years ago I remember trying to have a sane discussion about “non blocking” and I remember saying “something” will block eventually no matter what… anything from the buffer being full on the NIC to your cpu being at anything less than 100%. Does it shake out to any real advantage?

> What is the virtual thread / event loop pattern seeking to optimize? Is it context switching?

Throughput.

Some workloads are not CPU-bound or memory-bound, and spend the bulk of their time waiting for external processes to make data available.

If your workloads are expected to stay idle while waiting for external events, you can switch to other tasks while you wait for those external events to trigger.

This is particularly convenient if the other tasks you're hoping to run are also tasks that are bound to stay idle while waiting for external events.

One of the textbook scenarios that suits this pattern well is making HTTP requests. Another one is request handlers, such as the controller pattern used so often in HTTP servers.

Perhaps the poster child of this pattern is Node.js. It might not be the performance king and might be single-threaded, but it features in the top spots in performance benchmarks such as TechEmpower's. Node.js is also highly favoured in function-as-a-service applications, as it's event-driven architecture is well suited for applications involving a hefty dose of network calls running on memory- and CPU-constrained systems.

Re: Java Virtual Threads: A Case Study

#22

Post by dikei »

hashmash wrote: post #5
kevingadd wrote: post #3

Earlier quoted context omitted.

One of the main reasons to do virtual threads is that it allows you to write naive "thread per request" code and still scale up significantly without hitting the kind of scaling limits you would with OS threads.

The problem with the naïve design is that even with virtual threads, you risk running out of (heap) memory if the threads ever block. Each task makes a bit of progress, allocates some objects, and then lets another one do the same thing. With virtual threads, you can limit the damage by using a semaphore, but you still need to tune the size. This isn't much different than sizing a traditional thread pool, and so I'm…

> The problem with the naïve design is that even with virtual threads, you risk running out of (heap) memory if the threads ever block.

The key with virtual threads is they are so light weight that you can have thousands of them running concurrently: even when they block for I/O, it doesn't matter. It's similar to light weight coroutine in other language like Go or Kotlin.

Re: Java Virtual Threads: A Case Study

#23

Post by groestl »

hashmash wrote: post #15
initplus wrote: post #11

Earlier quoted context omitted.

The benefits from virtual threads come from the simple API that it presents to the programmer. It's not a performance optimization.

But that same benefit was always available with platform threads -- a simple API. What is the real gain by using virtual threads? It's either going to be performance or memory utilization.

It's combining the benefits from async models (state machines separated from os threads, thus more optimal for I/O bound workload), with the benefits from proper threading models (namely the simpler human interface).

Memory utilization & performance is going to be similar to the async callback mess.

Re: Java Virtual Threads: A Case Study

#24

Post by bheadmaster »

nlitened wrote: post #10
fzeindl wrote: post #4

Earlier quoted context omitted.

Does it shake out to any real advantage? To put it shortly: Writing single-threaded blocking code is far easier for most people and has many other benefits, like more understandable and readable programs: https://www.youtube.com/watch?v=449j7oKQVkc The main reason why non-blocking IO with it's style of intertwining concurrency and algorithms came along is that starting a thread for every request was too expensive. Wi…

> is far easier for most people I’d say that writing single-threaded code is far easier for _all_ people, even async code experts :) Also, single-threaded code is supported by programming language facilities: you have a proper call stack, thread-local vars, exceptions bubbling up, structured concurrency, simple resource management (RAII, try-with-resources, defer). Easy to reason and debug on language level. Async ru…

> Async runtimes are always complicated, filled with leaky abstractions, it’s like another language that one has to learn in addition, but with a less thought-out, ad-hoc design. Difficult to reason and debug, especially in edge cases

Async runtimes themselves are simply attempts to bolt-on green threads on top of a language that doesn't support them on a language level. In JavaScript, async/await uses Promises to enable callback-code to interact with key language features like try/catch, for/while/break, return, etc. In Python, async/await is just syntax sugar for coroutines, which are again just syntax sugar for CPS-style classes with methods split at each "yield". Not sure about Rust, but it probably also uses some Rust macro magic to do something similar.

Re: Java Virtual Threads: A Case Study

#25

Post by chipdart »

fzeindl wrote: post #4
exabrial wrote: post #2

What is the virtual thread / event loop pattern seeking to optimize? Is it context switching? A number of years ago I remember trying to have a sane discussion about “non blocking” and I remember saying “something” will block eventually no matter what… anything from the buffer being full on the NIC to your cpu being at anything less than 100%. Does it shake out to any real advantage?

Does it shake out to any real advantage? To put it shortly: Writing single-threaded blocking code is far easier for most people and has many other benefits, like more understandable and readable programs: https://www.youtube.com/watch?v=449j7oKQVkc The main reason why non-blocking IO with it's style of intertwining concurrency and algorithms came along is that starting a thread for every request was too expensive. Wi…

> To put it shortly: Writing single-threaded blocking code is far easier for most people and has many other benefits, like more understandable and readable programs:

I think you're missing the whole point.

The reason why so many smart people invest their time on "virtual threads" is developer experience. The goal is to turn writing event-driven concurrent code into something that's as easy as writing single-threaded blocking code.

Check why C#'s async/await implementation is such a huge success and replaced all past approaches overnight. Check why node.js is such a huge success. Check why Rust's async support is such a hot mess. It's all about developer experience.

Re: Java Virtual Threads: A Case Study

#26

Post by taspeotis »

My rough understanding is that this is similar to async/await in .NET?

It’s a shame this article paints a neutral (or even negative) experience with virtual threads.

We rewrote a boring CRUD app that spent 99% of its time waiting the database to respond to be async/await from top-to-bottom. CPU and memory usage went way down on the web server because so many requests could be handled by far fewer threads.

Re: Java Virtual Threads: A Case Study

#27

Post by hashmash »

groestl wrote: post #23
hashmash wrote: post #15

Earlier quoted context omitted.

But that same benefit was always available with platform threads -- a simple API. What is the real gain by using virtual threads? It's either going to be performance or memory utilization.

It's combining the benefits from async models (state machines separated from os threads, thus more optimal for I/O bound workload), with the benefits from proper threading models (namely the simpler human interface). Memory utilization & performance is going to be similar to the async callback mess.

Why is an async model better than using OS threads for an I/O bound workload? The OS is doing async stuff internally and shielding the complexity with threads. With virtual threads this work has shifted to the JVM. Can the JVM do threads better than the OS?

Re: Java Virtual Threads: A Case Study

#28

Post by gregopet »

immibis wrote: post #8
gregopet wrote: post #7

Earlier quoted context omitted.

It's a brave attempt to release the programmer from worrying or even thinking about thread pools and blocking code. Java has gone all in - they even cancelled a non-blocking rewrite of their database driver architecture because why have that if you won't have to worry about blocking code? And the JVM really is a marvel of engineering, it's really really good at what it does, so what team to better pull this off? So f…

So... What is it seeking to optimize? Why did you need a thread pool before but not any more? What resource was exhausted to prevent you from putting every request on a thread?

It's mainly trying to make you not worry about how many threads you create (and not worry about the caveats that come with optimising how many threads you create, which is something you are very often forced to do).

You can create a thread in your code and not worry whether that thing will then be some day run in a huge loop or receive thousands of requests and therefore spend all your memory on thread overhead. Go and other languages (in Java's ecosystem there's Kotlin for example) employ similar mechanisms to avoid native thread overhead, but you have to think about them. Like, there's tutorial code where everything is nice & simple, and then there's real world code where a lot of it must run in these special constructs that may have little to do with what you saw in those first "Hello, world" samples.

Java's approach tries to erase the difference between virtual and real threads. The programmer should have to employ no special techniques when using virtual threads and should be able to use everything the language has to offer (this isn't true in many languages' virtual/green threads implementations). Old libraries should continue working and perhaps not even be aware they're being run on virtual threads (although, caveats do apply for low level/high performance stuff, see above posts). And libraries that you interact with don't have to care what "model" of green threading you're using or specifically expose "red" and "blue" functions.

Re: Java Virtual Threads: A Case Study

#29

Post by chipdart »

immibis wrote: post #8
gregopet wrote: post #7

Earlier quoted context omitted.

It's a brave attempt to release the programmer from worrying or even thinking about thread pools and blocking code. Java has gone all in - they even cancelled a non-blocking rewrite of their database driver architecture because why have that if you won't have to worry about blocking code? And the JVM really is a marvel of engineering, it's really really good at what it does, so what team to better pull this off? So f…

So... What is it seeking to optimize? Why did you need a thread pool before but not any more? What resource was exhausted to prevent you from putting every request on a thread?

> So... What is it seeking to optimize?

The goal is to maximize the number of tasks you can run concurrently, while imposing on the developers a low cognitive load to write and maintain the code.

> Why did you need a thread pool before but not any more?

You still need a thread pool. Except with virtual threads you are no longer bound to run a single task per thread. This is specially desirable when workloads are IO-bound and will expectedly idle while waiting for external events. If you have a never-ending queue of tasks waiting to run, why should you block a thread consuming that task queue by running a task that stays idle while waiting for something to happen? You're better off starting the task and setting it aside the moment it awaits for something to happen.

> What resource was exhausted to prevent you from putting every request on a thread?

Re: Java Virtual Threads: A Case Study

#30

Post by jsiepkes »

taspeotis wrote: post #26

My rough understanding is that this is similar to async/await in .NET? It’s a shame this article paints a neutral (or even negative) experience with virtual threads. We rewrote a boring CRUD app that spent 99% of its time waiting the database to respond to be async/await from top-to-bottom. CPU and memory usage went way down on the web server because so many requests could be handled by far fewer threads.

> My rough understanding is that this is similar to async/await in .NET?

Well somewhat but also not really. They are green threads like async/await, but it's use is more transparent, unlike async/await.

So there are no special "async methods". You just instantiate a "VirtualThread" where you normally instantiate a (kernel) "Thread" and then use it like any other (kernel) thread. This works because for example all blocking IO API will be automatically converted to non-blocking IO underwater.

Post reply on HN