Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Java Virtual Threads: A Case Study

infoq.com

Post reply on HN
Print view View original

81–90 of 199 posts

Re: Java Virtual Threads: A Case Study

#81

Post by mike_hearn »

delusional wrote: post #40
devjab wrote: post #37

Earlier quoted context omitted.

> My rough understanding is that this is similar to async/await in .NET? Not really. What C# does is sort of similar but it has the disadvantages of splitting your code ecosystem into non-blocking/blocking code. This means you can “accidentally” start your non-blocking code. Something which may cause your relatively simple API to consume a ridiculous amount of resources. It also makes it much more complicated to upda…

From what I recall, and this is a while ago so bare with me, Java Virtual Threads still have a lot of pitfalls where the promise of concurrency isn't really fulfilled. I seem to remember that is was some pretty basic operations (like maybe read or something) that caused the thread not to unmount, and therefore just block the underlying os thread. At that point you've just invented the world's most complicated thread…

Reading from sockets definitely works. It'd be pretty useless if it didn't.

Some operations that don't cause a task switch to another virtual thread are:

- If you've called into a native library and back into Java that then blocks. In practice this never happens because Java code doesn't rely on native libraries or frameworks that much and when it does happen it's nearly always in-and-out quickly without callbacks. This can't be fixed by the JVM, however.

- File IO. No fundamental problem here, it can be fixed, it's just that not so many programs need tens of thousands of threads doing async file IO.

- If you're holding a lock using 'synchronized'. No fundamental problem here, it's just annoying because of how HotSpot is implemented. They're fixing this at the moment.

In practice it's mostly the last one that causes issues in real apps. It's not hard to work around, and eventually those workarounds won't be needed anymore.

Re: Java Virtual Threads: A Case Study

#82

Post by segfaltnh »

fulafel wrote: post #80
taspeotis wrote: post #26

My rough understanding is that this is similar to async/await in .NET? It’s a shame this article paints a neutral (or even negative) experience with virtual threads. We rewrote a boring CRUD app that spent 99% of its time waiting the database to respond to be async/await from top-to-bottom. CPU and memory usage went way down on the web server because so many requests could be handled by far fewer threads.

Can you expand on how the benefit in your rewrite came about? Threads don't consume CPU when they're waiting for the DB, after all. And threads share memory with each other. (I guess scaling to ridiculous levels you could be approaching trouble if you have O(100k) outstanding DB queries per application server, hope you have a DB that can handle millions of oustanding DB queries then!)

In large numbers the cost of switching between threads does consume CPU while they're waiting for the database. This is why green threads exist, to have large numbers of in flight work executing over a smaller number of OS threads.

Re: Java Virtual Threads: A Case Study

#83

Post by brabel »

vbezhenar wrote: post #75
LinXitoW wrote: post #57

From my very limited exposure to virtual threads and the older solution (thread pools), the biggest hurdle was the extensive use of ThreadLocals by most popular libraries. In one project I had to basically turn a reactive framework into a one thread per request framework, because passing around the MDC (a kv map of extra logging information) was a horrible pain. Getting it to actually jump ship from thread to thread…

What do you mean by hurdle? ThreadLocals work just fine with virtual threads.

It's not recommended though.

See https://openjdk.org/jeps/429

If you keep ThreadLocal variables, they get inherited by child Threads. If you make many thousands of them, the memory footprint becomes completely unacceptable. If the memory used by ThreadLocal variables is large, it also makes it more expensive to create new Threads (virtual or not), so you lose most advantages of Virtual Threads by doing that.

Re: Java Virtual Threads: A Case Study

#84

Post by mike_hearn »

immibis wrote: post #70
gregopet wrote: post #28

Earlier quoted context omitted.

It's mainly trying to make you not worry about how many threads you create (and not worry about the caveats that come with optimising how many threads you create, which is something you are very often forced to do). You can create a thread in your code and not worry whether that thing will then be some day run in a huge loop or receive thousands of requests and therefore spend all your memory on thread overhead. Go a…

It seems that the answer to the question was "memory". Stack allocations, presumably. You have answered by telling us that virtual threads are better than real threads because real threads suck, but you didn't say why they suck or why virtual threads don't suck in the same way.

Real threads don't suck but they pay a price for generality. The kernel doesn't know what software you're going to run, and there's no standards for how that software might use the stack. So the kernel can't optimize by making any assumptions.

Virtual threads are less general than kernel threads. If you use a virtual thread to call out of the JVM you lose their benefits, because the JVM becomes like the kernel and can't make any assumptions about the stack.

But if you are running code controlled by the JVM, then it becomes possible to do optimizations (mostly stack related) that otherwise can't be done, because the GC and the compiler and the threads runtime are all developed together and work together.

Specifically, what HotSpot can do moving stack frames to and from the heap very fast, which interacts better with the GC. For instance if a virtual thread resumes, iterates in a loop and suspends again, then the stack frames are never copied out of the heap onto the kernel stack at all. Hotspot can incrementally "pages" stack frames out of the heap. Additionally, the storage space used for a suspended virtual thread stack is a lot smaller than a suspended kernel stack because a lot of administrative goop doesn't need to be saved at all.

Re: Java Virtual Threads: A Case Study

#85

Post by brabel »

bberrry wrote: post #62
LinXitoW wrote: post #57

From my very limited exposure to virtual threads and the older solution (thread pools), the biggest hurdle was the extensive use of ThreadLocals by most popular libraries. In one project I had to basically turn a reactive framework into a one thread per request framework, because passing around the MDC (a kv map of extra logging information) was a horrible pain. Getting it to actually jump ship from thread to thread…

If you are already in a reactive framework, why would you change to virtual threads? Those frameworks pool threads and have their own event loop so I would say they are not suitable for virtual thread migration.

Yes, if you're happy with the reactive frameworks there's no reason to migrate. Most people, however, would love to remove their complexities from their code bases. Virtual Threads are much, much easier to program with. There's downsides, like not being able to easily limit concurrency, having to implement your own timeout mechanisms etc. but that will probably be provided by a common lib sooner or later which hopefully provides identical features to reactive frameworks, while being much, much simpler.

Re: Java Virtual Threads: A Case Study

#86

Post by troupo »

devjab wrote: post #37
taspeotis wrote: post #26

My rough understanding is that this is similar to async/await in .NET? It’s a shame this article paints a neutral (or even negative) experience with virtual threads. We rewrote a boring CRUD app that spent 99% of its time waiting the database to respond to be async/await from top-to-bottom. CPU and memory usage went way down on the web server because so many requests could be handled by far fewer threads.

> My rough understanding is that this is similar to async/await in .NET? Not really. What C# does is sort of similar but it has the disadvantages of splitting your code ecosystem into non-blocking/blocking code. This means you can “accidentally” start your non-blocking code. Something which may cause your relatively simple API to consume a ridiculous amount of resources. It also makes it much more complicated to upda…

Erlang, not Go, should be the obvious choice for concurrency, but it's impossible to retrofit Erlang's concurrency onto existing systems.

Re: Java Virtual Threads: A Case Study

#87

Post by haspok »

frevib wrote: post #76
haspok wrote: post #73

Earlier quoted context omitted.

Just a side note, async JDBC was a thing way before Loom came about, and it failed miserably. I'm not sure why, but my guess would be is that most enterprise software is not web-scale, so JDBC worked well as it was. Also, all the database vendors provided their drivers implementing the JDBC API - good luck getting Oracle or IBM contribute to R2DBC.. (Actually, I stand corrected: there is an Oracle R2DBC driver now -…

It could also be that there just isn’t enough demand for a non-blocking JDBC. For example, Postgresql server is not coping very well with lots of simultaneous connections, due to it’s (a.o.) process-per-connection model. From the client-side (JDBC), a small thread poool would be enough to max out the Postgresql server. And there is almost no benefit of using non-blocking vs a small thread pool.

I would argue the main benefit would be that the threadpool that the developer would create anyway would instead be created by the async database driver, which has more intimate knowledge about the server's capabilities. Maybe it knows the limits to the number of connections, or can do other smart optimizations. In any case, for the developer it would be a more streamlined experience, with less code needed, and better defaults.

Re: Java Virtual Threads: A Case Study

#88

Post by brabel »

immibis wrote: post #70
gregopet wrote: post #28

Earlier quoted context omitted.

It's mainly trying to make you not worry about how many threads you create (and not worry about the caveats that come with optimising how many threads you create, which is something you are very often forced to do). You can create a thread in your code and not worry whether that thing will then be some day run in a huge loop or receive thousands of requests and therefore spend all your memory on thread overhead. Go a…

It seems that the answer to the question was "memory". Stack allocations, presumably. You have answered by telling us that virtual threads are better than real threads because real threads suck, but you didn't say why they suck or why virtual threads don't suck in the same way.

OS Threads do not suck, they're great. But they are expensive to create as they require a syscall, and they're expensive to maintain as they consume quite a bit of memory just to exist, even if you don't need it (due to how they must pre-allocate a stack which apparently is around 2MB initially, and can't be made smaller as in most cases you will need even more, so it would make most cases worse).

Virtual Threads are very fast to create and allocate only the memory needed by the actual call stack, which can be much less than for OS Threads.

Also, blocking code is very simple compared to the equivalent async code. So using blocking code makes your code much easier to follow. Check out examples of reactive frameworks for Java and you will quickly understand why.

Re: Java Virtual Threads: A Case Study

#89

Post by imtringued »

hashmash wrote: post #5
kevingadd wrote: post #3

Earlier quoted context omitted.

One of the main reasons to do virtual threads is that it allows you to write naive "thread per request" code and still scale up significantly without hitting the kind of scaling limits you would with OS threads.

The problem with the naïve design is that even with virtual threads, you risk running out of (heap) memory if the threads ever block. Each task makes a bit of progress, allocates some objects, and then lets another one do the same thing. With virtual threads, you can limit the damage by using a semaphore, but you still need to tune the size. This isn't much different than sizing a traditional thread pool, and so I'm…

What you are complaining about has nothing to do with thread pools or virtual threads. You're pointing out the fact that more parallelism will also need more hardware and that a finite hardware budget will need a back pressure strategy to keep resource consumption within a limit. While you might be correct that "sizing a traditional thread pool" is a back pressure strategy that can be applied to virtual threads, the problem with it is that IO bound threads will prevent CPU bound threads from making progress. You don't want to apply back pressure based on the number of tasks. You want back pressure to be in response to resource utilization, so that enough tasks get scheduled to max out the hardware.

This is a common problem with people using Java parallel streams, because they by default share a single global thread pool and the way to use your own thread pool is also extremely counterintuitive, because it essentially relies on some implicit thread local magic to choose to distribute the stream in the thread pool that the parallel stream was launched on, instead of passing it as a parameter.

It would be best if people came up with more dynamic back pressure strategies, because this is a more general problem that goes way beyond thread pools. In fact, one of the key problems of automatic parallelization is deciding at what point there is too much parallelization.

Re: Java Virtual Threads: A Case Study

#90

Post by frevib »

haspok wrote: post #87
frevib wrote: post #76

Earlier quoted context omitted.

It could also be that there just isn’t enough demand for a non-blocking JDBC. For example, Postgresql server is not coping very well with lots of simultaneous connections, due to it’s (a.o.) process-per-connection model. From the client-side (JDBC), a small thread poool would be enough to max out the Postgresql server. And there is almost no benefit of using non-blocking vs a small thread pool.

I would argue the main benefit would be that the threadpool that the developer would create anyway would instead be created by the async database driver, which has more intimate knowledge about the server's capabilities. Maybe it knows the limits to the number of connections, or can do other smart optimizations. In any case, for the developer it would be a more streamlined experience, with less code needed, and bette…

I think we’re confusing async and non-blocking? Non-blocking is the part what makes virtual threads more efficient than threads. Async is the programming style; e.g. do things concurrently. Async can be implemented with threads or non-blocking, if the API supports it. I was merely arguing that a non-blocking JDBC has little merit as the connections to a DB are limited. Non-blocking APIs are only beneficial when there are lots, > 10k connections.

JDBC knows nothing about the amount of connections a server can handle, but to try so many connections until it won’t connect any more.

| In any case, for the developer it would be a more streamlined experience, with less code needed, and better defaults.

I agree it would be best not to bother the dev with what is going on under the hood.

Post reply on HN