Live data from Hacker News

Java Virtual Threads: A Case Study

infoq.com

91–100 of 199 posts

Re: Java Virtual Threads: A Case Study

#91
post #65

Earlier quoted context omitted.

It should be pointed out, that the main reason they didn't go further was because of added complexity in .NET, when async/await already exists. > Green threads introduce a completely new async programming model. The interaction between green threads and the existing async model is quite complex for .NET developers. For example, invoking async methods from green thread code requires a sync-over-async code pattern that…

This FAQ is a bit outdated in places, and is not something most users should worry about in practice. JVM Green Threads here serve predominantly back-end scenarios, where most of the items on the list are not of concern. This list also exists to address bad habits that carried over from before the tasks were introduced, many years ago. In general, the perceived want of green threads is in part caused by misunderstand…

I don't think that this would be a good showcase for Virtual Threads. The "async" API for Java is CompletableFutures, right? thats been stable for something like 10 years, so no real change since Java 8.

You'd jsut have to define a ThreadPool with n Threads before, where each request would've blocked one pending thread. Now it just keeps going.

So your equivalent Java example should've been something like this, but again: the completeable futures api is pretty old at this point.

    @HttpExchange(value = "https://news.ycombinator.com")
    interface HnClient {
        @GetExchange("news?p={page}")
        CompletableFuture getNews(@PathVariable("page") Integer page);
    }

    @RequiredArgsConstructor
    @Service
    class HnService {
        private final HnClient hnClient;
        List getNews() {
            var requests = IntStream.rangeClosed(1, 4)
                                    .boxed().map(hnClient::getNews).toList();
            return requests.stream().map(CompletableFuture::join).toList();
        }
    }

Re: Java Virtual Threads: A Case Study

#92
post #80

Earlier quoted context omitted.

Can you expand on how the benefit in your rewrite came about? Threads don't consume CPU when they're waiting for the DB, after all. And threads share memory with each other. (I guess scaling to ridiculous levels you could be approaching trouble if you have O(100k) outstanding DB queries per application server, hope you have a DB that can handle millions of oustanding DB queries then!)

In large numbers the cost of switching between threads does consume CPU while they're waiting for the database. This is why green threads exist, to have large numbers of in flight work executing over a smaller number of OS threads.

When using OS threads, there's no switching when they are waiting for a socket (db connection). The OS knows to wake the thread up only when there's something new to see on the connection.

Re: Java Virtual Threads: A Case Study

#93
post #53

Earlier quoted context omitted.

You will still have to worry, too many virtual threads will imply too much context switching. However, virtual threads will be always interruptable on I/O, as they are not mapped to actual o.s. threads, but rather simulated by the JVM which will executed a number of instructions for each virtual thread. This gives the chance to the JVM to use real threads more efficiently, avoiding that threads remain unused while wa…

What do you mean by context switching? My understanding is that virtual threads mostly eliminate context switching - for N CPUs JVM creates N platform threads and they run virtual threads as needed. There is no real context switching apart from GC and other JVM internal threads. A platform thread picking another virtual thread to run after its current virtual thread is blocked on IO is not a context switch, that is a…

The JVM will need to do context switching when reallocating the real thread that is running a blocked virtual thread to the next available virtual thread. It won't be CPU context switching, but context switching happens at the JVM level and represents an effort.

Re: Java Virtual Threads: A Case Study

#94
post #83

Earlier quoted context omitted.

What do you mean by hurdle? ThreadLocals work just fine with virtual threads.

It's not recommended though. See https://openjdk.org/jeps/429 If you keep ThreadLocal variables, they get inherited by child Threads. If you make many thousands of them, the memory footprint becomes completely unacceptable. If the memory used by ThreadLocal variables is large, it also makes it more expensive to create new Threads (virtual or not), so you lose most advantages of Virtual Threads by doing that.

I don't think that's correct. ThreadLocals should behave just like on regular OS threads, the difference is that you can suddenly create millions of them.

You used to be able to depend on OS threads getting reused because you were pooling them. You can do the same with virtual threads if you wish and you will get the same behavior. The difference is we ought to spawn new threads per task now.

Side note, you have to specifically use InheritableThreadLocal to get the inheritance behavior you speak of.

Re: Java Virtual Threads: A Case Study

#96
post #93

Earlier quoted context omitted.

What do you mean by context switching? My understanding is that virtual threads mostly eliminate context switching - for N CPUs JVM creates N platform threads and they run virtual threads as needed. There is no real context switching apart from GC and other JVM internal threads. A platform thread picking another virtual thread to run after its current virtual thread is blocked on IO is not a context switch, that is a…

The JVM will need to do context switching when reallocating the real thread that is running a blocked virtual thread to the next available virtual thread. It won't be CPU context switching, but context switching happens at the JVM level and represents an effort.

Ok. This JVM-level switching is called mounting/un-mounting of the virtual thread and is supposed to be several orders of magnitude cheaper compared to normal context switch. You should be fine with millions of virtual threads.

Re: Java Virtual Threads: A Case Study

#97
post #71

Earlier quoted context omitted.

What do you mean by context switching? My understanding is that virtual threads mostly eliminate context switching - for N CPUs JVM creates N platform threads and they run virtual threads as needed. There is no real context switching apart from GC and other JVM internal threads. A platform thread picking another virtual thread to run after its current virtual thread is blocked on IO is not a context switch, that is a…

"they run virtual threads as needed" - so when one virtual thread is no longer needed and another one is needed, they switch context, yes?

This is called mounting/un-mounting and is much cheaper than a context switch.

Re: Java Virtual Threads: A Case Study

#98
post #91

Earlier quoted context omitted.

This FAQ is a bit outdated in places, and is not something most users should worry about in practice. JVM Green Threads here serve predominantly back-end scenarios, where most of the items on the list are not of concern. This list also exists to address bad habits that carried over from before the tasks were introduced, many years ago. In general, the perceived want of green threads is in part caused by misunderstand…

I don't think that this would be a good showcase for Virtual Threads. The "async" API for Java is CompletableFutures, right? thats been stable for something like 10 years, so no real change since Java 8. You'd jsut have to define a ThreadPool with n Threads before, where each request would've blocked one pending thread. Now it just keeps going. So your equivalent Java example should've been something like this, but a…

Structured concurrency is still being developed: https://openjdk.org/jeps/453

Also, I wouldnt consider that the equivalent Java code. That is all Spring and Lombok magic. Just write the code and just use java.net.HttpClient.

Re: Java Virtual Threads: A Case Study

#99

Earlier quoted context omitted.

The benefits from virtual threads come from the simple API that it presents to the programmer. It's not a performance optimization.

But that same benefit was always available with platform threads -- a simple API. What is the real gain by using virtual threads? It's either going to be performance or memory utilization.

Create 100k platform threads and you'll find out.

Re: Java Virtual Threads: A Case Study

#100

Earlier quoted context omitted.

Yeah, and it's generally good to be RAM limited instead of CPU, no? The alternative is blowing a bunch of time on syscalls and OS scheduler overhead. Also the virtual threads run on a "traditional" thread pool to my understanding, so you can just tweak the number of worker threads to cap the total concurrency. The benefit is it's overall more efficient (in the general case) and lets you write linear blocking code (as…

The OS scheduler is still there (for the carrier threads), but now you've added on top of that FJ pool based scheduler overhead. Although virtual threads don't have the syscall overhead when they block, there's a new cost caused by allocating the internal continuation object, and copying state into it. This puts more pressure on the garbage collector. Context switching cost due to CPU cache thrashing doesn't go away…

The study says there's surprising performance problems with Java's virtual thread implementation. Their test of throughput was also hilarious, they put 2000 OS threads vs 2000 virtual threads: most of the time OS threads don't start falling apart until 100k+ threads. You can architect an application such that you can handle 200k simultaneous connections using platform-thread-per-core, but it's harder to reason about than the linear, blocking code that virtual threads and async allow for.

> Context switching cost due to CPU cache thrashing doesn't go away regardless of which type of thread you're using.

Except it's not a context switch? You're jumping to another instruction in the program, one that should be very predictable. You might lose your cache, but it will depend on a ton of factors.

> there's a new cost caused by allocating the internal continuation object, and copying state into it.

This is more of a problem with the implementation (not every virtual thread language does it this way), but yeah this is more overhead on the application. I assume there's improvements that can be made to ease GC pressure, like using object pools.

Usually virtual threads are a memory vs CPU tradeoff that you typically use in massively concurrent IO-bound applications. Total throughput should take over platform threads with hundreds of thousands of connections, but below that probably perform worse, I'm not that surprised by that.

Post reply on HN