Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Java Virtual Threads: A Case Study

infoq.com

Post reply on HN
Print view View original

71–80 of 199 posts

Re: Java Virtual Threads: A Case Study

#71

Post by immibis »

frant-hartm wrote: post #55
giamma wrote: post #53

Earlier quoted context omitted.

You will still have to worry, too many virtual threads will imply too much context switching. However, virtual threads will be always interruptable on I/O, as they are not mapped to actual o.s. threads, but rather simulated by the JVM which will executed a number of instructions for each virtual thread. This gives the chance to the JVM to use real threads more efficiently, avoiding that threads remain unused while wa…

What do you mean by context switching? My understanding is that virtual threads mostly eliminate context switching - for N CPUs JVM creates N platform threads and they run virtual threads as needed. There is no real context switching apart from GC and other JVM internal threads. A platform thread picking another virtual thread to run after its current virtual thread is blocked on IO is not a context switch, that is a…

"they run virtual threads as needed" - so when one virtual thread is no longer needed and another one is needed, they switch context, yes?

Re: Java Virtual Threads: A Case Study

#72

Post by immibis »

davidgay wrote: post #16
immibis wrote: post #8

Earlier quoted context omitted.

So... What is it seeking to optimize? Why did you need a thread pool before but not any more? What resource was exhausted to prevent you from putting every request on a thread?

A thread per request has a high risk of overcommitting on CPU use, leading to a different set of problems. Virtual threads are scheduled on a fixed-size (based on number of cores) underlying (non-virtual) thread pool to avoid this problem.

Why can't virtual threads overcommit CPU use? If I have 4 CPUs and 4000 virtual threads running CPU-bound code, is that not overcommit? A system without overcommit would refuse to create the 5th thread.

Re: Java Virtual Threads: A Case Study

#73

Post by haspok »

gregopet wrote: post #7
exabrial wrote: post #2

What is the virtual thread / event loop pattern seeking to optimize? Is it context switching? A number of years ago I remember trying to have a sane discussion about “non blocking” and I remember saying “something” will block eventually no matter what… anything from the buffer being full on the NIC to your cpu being at anything less than 100%. Does it shake out to any real advantage?

It's a brave attempt to release the programmer from worrying or even thinking about thread pools and blocking code. Java has gone all in - they even cancelled a non-blocking rewrite of their database driver architecture because why have that if you won't have to worry about blocking code? And the JVM really is a marvel of engineering, it's really really good at what it does, so what team to better pull this off? So f…

Just a side note, async JDBC was a thing way before Loom came about, and it failed miserably. I'm not sure why, but my guess would be is that most enterprise software is not web-scale, so JDBC worked well as it was.

Also, all the database vendors provided their drivers implementing the JDBC API - good luck getting Oracle or IBM contribute to R2DBC.. (Actually, I stand corrected: there is an Oracle R2DBC driver now - it was released fairly recently though.)

EDIT: "failed miserably" is maybe too strong - but R2DBC certainly doesn't have the support and acceptance of JDBC.

Re: Java Virtual Threads: A Case Study

#74

Post by frevib »

exabrial wrote: post #2

What is the virtual thread / event loop pattern seeking to optimize? Is it context switching? A number of years ago I remember trying to have a sane discussion about “non blocking” and I remember saying “something” will block eventually no matter what… anything from the buffer being full on the NIC to your cpu being at anything less than 100%. Does it shake out to any real advantage?

They indeed optimize thread context switching. Taking the thread on and off the CPU is becoming expensive when there are thousands of threads.

You are right that everything blocks, even when going to L1 cache you have to wait 1 nanoseconds. But blocking in this context means waiting for “real” IO like a network request or spinning disk access. Virtual threads take away the problem that the thread sits there doing nothing for a while as it is waiting for data, before it is context switched.

Virtual threads won’t improve CPU-bound blocking. There the thread is actually occupying the CPU, so there is no problem of the thread doing nothing as with IO-bound blocking.

Re: Java Virtual Threads: A Case Study

#75

Post by vbezhenar »

LinXitoW wrote: post #57

From my very limited exposure to virtual threads and the older solution (thread pools), the biggest hurdle was the extensive use of ThreadLocals by most popular libraries. In one project I had to basically turn a reactive framework into a one thread per request framework, because passing around the MDC (a kv map of extra logging information) was a horrible pain. Getting it to actually jump ship from thread to thread…

What do you mean by hurdle? ThreadLocals work just fine with virtual threads.

Re: Java Virtual Threads: A Case Study

#76

Post by frevib »

haspok wrote: post #73
gregopet wrote: post #7

Earlier quoted context omitted.

It's a brave attempt to release the programmer from worrying or even thinking about thread pools and blocking code. Java has gone all in - they even cancelled a non-blocking rewrite of their database driver architecture because why have that if you won't have to worry about blocking code? And the JVM really is a marvel of engineering, it's really really good at what it does, so what team to better pull this off? So f…

Just a side note, async JDBC was a thing way before Loom came about, and it failed miserably. I'm not sure why, but my guess would be is that most enterprise software is not web-scale, so JDBC worked well as it was. Also, all the database vendors provided their drivers implementing the JDBC API - good luck getting Oracle or IBM contribute to R2DBC.. (Actually, I stand corrected: there is an Oracle R2DBC driver now -…

It could also be that there just isn’t enough demand for a non-blocking JDBC. For example, Postgresql server is not coping very well with lots of simultaneous connections, due to it’s (a.o.) process-per-connection model. From the client-side (JDBC), a small thread poool would be enough to max out the Postgresql server. And there is almost no benefit of using non-blocking vs a small thread pool.

Re: Java Virtual Threads: A Case Study

#77

Post by vbezhenar »

haspok wrote: post #73
gregopet wrote: post #7

Earlier quoted context omitted.

It's a brave attempt to release the programmer from worrying or even thinking about thread pools and blocking code. Java has gone all in - they even cancelled a non-blocking rewrite of their database driver architecture because why have that if you won't have to worry about blocking code? And the JVM really is a marvel of engineering, it's really really good at what it does, so what team to better pull this off? So f…

Just a side note, async JDBC was a thing way before Loom came about, and it failed miserably. I'm not sure why, but my guess would be is that most enterprise software is not web-scale, so JDBC worked well as it was. Also, all the database vendors provided their drivers implementing the JDBC API - good luck getting Oracle or IBM contribute to R2DBC.. (Actually, I stand corrected: there is an Oracle R2DBC driver now -…

R2DBC allows to efficiently maintain millions of connections to the database. But what database supports millions of connections? Not postgres for sure, and probably no other conventional database. So using reactive JDBC driver makes little sense, if you're going to use 1000 connections, 1000 threads will do just fine and bring little overhead. Those who use Java, don't care about spending 100 more MB of RAM when their service already eats 60GB.

Re: Java Virtual Threads: A Case Study

#78

Post by mike_hearn »

anonymousDan wrote: post #66
frant-hartm wrote: post #55

Earlier quoted context omitted.

What do you mean by context switching? My understanding is that virtual threads mostly eliminate context switching - for N CPUs JVM creates N platform threads and they run virtual threads as needed. There is no real context switching apart from GC and other JVM internal threads. A platform thread picking another virtual thread to run after its current virtual thread is blocked on IO is not a context switch, that is a…

Does Java's implementation of virtual threads perform any kind of work stealing when a particular physical thread has no virtual threads to run (e.g. they are all blocked on I/O)?

It does. They get scheduled onto the ForkJoinPool which is a work stealing pool.

Re: Java Virtual Threads: A Case Study

#79

Post by neonsunset »

pjmlp wrote: post #65
peteri wrote: post #58

Earlier quoted context omitted.

It's a different model. Microsoft did work on green threads a while ago and decided against continuing. Links: https://github.com/dotnet/runtimelab/issues/2398 https://github.com/dotnet/runtimelab/blob/feature/green-thre...

It should be pointed out, that the main reason they didn't go further was because of added complexity in .NET, when async/await already exists. > Green threads introduce a completely new async programming model. The interaction between green threads and the existing async model is quite complex for .NET developers. For example, invoking async methods from green thread code requires a sync-over-async code pattern that…

This FAQ is a bit outdated in places, and is not something most users should worry about in practice.

JVM Green Threads here serve predominantly back-end scenarios, where most of the items on the list are not of concern. This list also exists to address bad habits that carried over from before the tasks were introduced, many years ago.

In general, the perceived want of green threads is in part caused by misunderstanding of that one bad article about function coloring. And that one bad article about function coloring also does not talk about the way you do async in C#.

Async/await in C# in back-end is a very easy to work with model with explicit understanding where a method returns an operation that promises to complete in the future or not, and composing tasks[0] for easy (massive) concurrency is significantly more idiomatic than doing so with green threads or completable futures that existed in Java before these. And as evidenced by adoption of green threads by large scale Java projects, turns out the failure modes share similarities except green threads end up violating way more expectations and the code author may not have any indication or explicit mechanism to address this, like using AsyncLocal.

Also one change to look for is "Runtime Handled Tasks" project in .NET that will replace Roslyn-generated state machine code with runtime-provided suspension mechanism which will only ever suspend at true suspension points where task's execution actually yields asynchronously. So far numbers show at least 5x decrease in overhead, which is massive and will bring performance of computation heavy async paths in line with sync ones:

https://github.com/dotnet/runtimelab/blob/feature/async2-exp...

Note that you were trivially able to have millions of scheduled tasks even before that as they are very lightweight.

[0]: e.g. sending requests in parallel is just this

    using var http = new HttpClient() {
        BaseAddress = new("https://news.ycombinator.com/news")
    };

    var requests = Enumerable
        .Range(1, 4)
        .Select(n => $"?p={n}")
        .Select(http.GetStringAsync);

    var pages = await Task.WhenAll(requests);

Re: Java Virtual Threads: A Case Study

#80

Post by fulafel »

taspeotis wrote: post #26

My rough understanding is that this is similar to async/await in .NET? It’s a shame this article paints a neutral (or even negative) experience with virtual threads. We rewrote a boring CRUD app that spent 99% of its time waiting the database to respond to be async/await from top-to-bottom. CPU and memory usage went way down on the web server because so many requests could be handled by far fewer threads.

Can you expand on how the benefit in your rewrite came about? Threads don't consume CPU when they're waiting for the DB, after all. And threads share memory with each other.

(I guess scaling to ridiculous levels you could be approaching trouble if you have O(100k) outstanding DB queries per application server, hope you have a DB that can handle millions of oustanding DB queries then!)

Post reply on HN