Java Virtual Threads: A Case Study • Hacker News Board

Re: Java Virtual Threads: A Case Study

#141

Post by tomp » Wed, Jul 17, 2024, 8:33 PM UTC

jayd16 wrote: post #137

tomp wrote: post #134

Earlier quoted context omitted.

They're not equivalent to Go's goroutines. Go's goroutines are preemptive (and Go's development team went through a lot of pain to make them such). Java's lightweight threads aren't. Java's repeating the same mistakes that Go made (and learned from) 10 years ago.

Virtual threads could be scheduled pre-emptively but currently the scheduler will wait for some kind of thread sleep to schedule another virtual thread. That's just a scheduler implementation detail and the spec is such that a time slice scheduler could be implemented.

Yes, but the problem is that the spec is such that preemptive blocking doesn't need to be implemented.

That means that Java programmers have to be very careful when writing code, lest they block the entire underlying (OS) thread!

Again, Go already went through that experience. It was painful. Java should have learned and implemented it from the start

Re: Java Virtual Threads: A Case Study

#142

Post by jayd16 » Wed, Jul 17, 2024, 8:34 PM UTC

he0001 wrote: post #128

jayd16 wrote: post #110

Earlier quoted context omitted.

There is overlap but they really don't solve the same problem. Cooperative threading has its own advantages and patterns that won't be served by virtual threads.

What patterns does async/await solve which virtual threads don’t?

If you need to be explicit about thread contexts because you're using a thread that's bound to some other runtime (say, a GL Context) or you simply want to use a single thread for synchronization like is common in UI programming with a Main/UI Thread, async/await does quite well. The async/await sugar ends up being a better devx than thread locking and implicit threading just doesn't cut it.

In Java they're working on a structured concurrency library to bridge this gap, but IMO, it'll end up looking like async/await with all its ups and downs but with less sugar.

Re: Java Virtual Threads: A Case Study

#143

Post by troupo » Wed, Jul 17, 2024, 8:36 PM UTC

immibis wrote: post #120

detinho wrote: post #105

Earlier quoted context omitted.

I think parent is saying overcommit with OS threads. 4k requests = 4k OS threads. That would lead to the problems parent is talking about.

Why wouldn't 4k virtual threads lead to the same problems?

Because they don't create 4k real threads, and can be scheduled on n=CPU Cores OS threads

Re: Java Virtual Threads: A Case Study

#144

Post by Someone » Wed, Jul 17, 2024, 8:45 PM UTC

kllrnohj wrote: post #126

brabel wrote: post #88

Earlier quoted context omitted.

OS Threads do not suck, they're great. But they are expensive to create as they require a syscall, and they're expensive to maintain as they consume quite a bit of memory just to exist, even if you don't need it (due to how they must pre-allocate a stack which apparently is around 2MB initially, and can't be made smaller as in most cases you will need even more, so it would make most cases worse). Virtual Threads are…

> and they're expensive to maintain as they consume quite a bit of memory just to exist, even if you don't need it (due to how they must pre-allocate a stack which apparently is around 2MB initially, I'm not familiar with windows, but this certainly isn't the case on Linux. It only costs 2mb-8mb of virtual address space, not actual physical memory. And there's no particular reason to believe the JVM can have a list o…

> The downside though is you don't have any preemption, which depending on your usage is a […] massive downside.

Nobody is taking OS threads away, so you can choose to use them when they better fit your use case.

Re: Java Virtual Threads: A Case Study

#145

Post by nimish » Wed, Jul 17, 2024, 9:23 PM UTC

jayd16 wrote: post #137

tomp wrote: post #134

Earlier quoted context omitted.

They're not equivalent to Go's goroutines. Go's goroutines are preemptive (and Go's development team went through a lot of pain to make them such). Java's lightweight threads aren't. Java's repeating the same mistakes that Go made (and learned from) 10 years ago.

Virtual threads could be scheduled pre-emptively but currently the scheduler will wait for some kind of thread sleep to schedule another virtual thread. That's just a scheduler implementation detail and the spec is such that a time slice scheduler could be implemented.

This is a really unfortunate gotcha that's not at all obvious. Does it kick preemption up a layer to the OS then?

Re: Java Virtual Threads: A Case Study

#146

Post by Jtsummers » Wed, Jul 17, 2024, 9:44 PM UTC

nimish wrote: post #145

jayd16 wrote: post #137

Earlier quoted context omitted.

Virtual threads could be scheduled pre-emptively but currently the scheduler will wait for some kind of thread sleep to schedule another virtual thread. That's just a scheduler implementation detail and the spec is such that a time slice scheduler could be implemented.

This is a really unfortunate gotcha that's not at all obvious. Does it kick preemption up a layer to the OS then?

The "not at all obvious" gotcha is described in the documentation near the top, under the heading "What is a Virtual Thread?":

https://docs.oracle.com/en/java/javase/21/core/virtual-threa...

> Like a platform thread, a virtual thread is also an instance of java.lang.Thread. However, a virtual thread isn't tied to a specific OS thread. A virtual thread still runs code on an OS thread. However, when code running in a virtual thread calls a blocking I/O operation, the Java runtime suspends the virtual thread until it can be resumed. The OS thread associated with the suspended virtual thread is now free to perform operations for other virtual threads.

It's not been hidden at all in their presentation on virtual threads.

The OS thread that the virtual thread is mounted to can still be preempted, but that won't free up the OS thread for another virtual thread. However, if you use them for what they're intended for this shouldn't be a problem. In practice, it will be because no one can be bothered to RTFM.

Re: Java Virtual Threads: A Case Study

#147

Post by kbolino » Wed, Jul 17, 2024, 9:53 PM UTC

whateveracct wrote: post #102

logicchains wrote: post #34

Earlier quoted context omitted.

>Async runtimes themselves are simply attempts to bolt-on green threads on top of a language that doesn't support them on a language level. Haskell supports async code while also supporting green threads on a language level, and the async code has most of the same issues as async code in any other languages.

What problems exactly? Haskell has a few things that imo it does better than most languages in this area: - All IO is non-blocking by default. - FFI support for interruptible. - Haskell threads can be preempted externally - this allows you to ensure they never leak. Vs a goroutine that can just spin forever if it doesn't explicitly yield. - There are various stdlib abstractions for building concurrent programs in a c…

> Haskell threads can be preempted externally - this allows you to ensure they never leak. Vs a goroutine that can just spin forever if it doesn't explicitly yield.

Goroutines are preemptible by the runtime (since https://go.dev/doc/go1.14#runtime) but they're still not addressable or killable through the language itself.

Re: Java Virtual Threads: A Case Study

#148

Post by written-beyond » Wed, Jul 17, 2024, 9:55 PM UTC

chipdart wrote: post #25

fzeindl wrote: post #4

Earlier quoted context omitted.

Does it shake out to any real advantage? To put it shortly: Writing single-threaded blocking code is far easier for most people and has many other benefits, like more understandable and readable programs: https://www.youtube.com/watch?v=449j7oKQVkc The main reason why non-blocking IO with it's style of intertwining concurrency and algorithms came along is that starting a thread for every request was too expensive. Wi…

> To put it shortly: Writing single-threaded blocking code is far easier for most people and has many other benefits, like more understandable and readable programs: I think you're missing the whole point. The reason why so many smart people invest their time on "virtual threads" is developer experience. The goal is to turn writing event-driven concurrent code into something that's as easy as writing single-threaded…

As someone who has written multiple productions services with Async Rust, that are under constant load, I disagree. I've had team members who have only written in C, pick up and start building very comprehensive and performant services in Rust in a matter of days.

How do you developers spew such strong opinions without taking a moment to think about what you're about to say. Rust cannot be directly compared to C#, Java or even Go.

You don't get a runtime or a GC with rust. The developer experience is excellent, you get a lot of control over everything you're building with it. Yes it's not as magical as languages and runtimes like you've mentioned, but the fact that I can at anytime rip those abstractions off and make my service extremely lightweight and performant is not something those languages will allow you to do.

And this is coming from someone who's written non blocking services before Async rust was a thing with just MIO.

The very fact Rust gets mentioned between these languages should be a tribute to the efforts of it's maintainers and core team. The amount of tooling and features they've added into the language gives developers of every realm liberty to try and build what they want.

Honestly, you can hold whatever opinion you want on any language but your comparison really doesn't make sense.

Re: Java Virtual Threads: A Case Study

#149

Post by jayceedenton » Wed, Jul 17, 2024, 10:00 PM UTC

pron wrote: post #63

Virtual threads do one thing: they allow creating lots of threads. This helps throughput due to Little's law [1]. But because this server here saturates the CPU with only a few threads (it doesn't do the fanout modern servers tend to do), this means that no significant improvements can be provided by virtual threads (or asynchronous programming, which operates on the same principle) while keeping everything else in t…

I guess at least their work has confirmed what we probably already knew intuitively: if you have CPU-intensive tasks, without waiting on anything, and you want to execute these concurrently, use traditional threads.

The advice "don't use virtual threads for that, it will be inefficient" really does need some evidence.

Mildly infuriating though that people may read this and think that somehow the JVM has problems in its virtual thread implementation. I admit their 'Unexpected findings' section is very useful work, but the moral of this story is: don't use virtual threads for this that they were not intended for. Use them when you want a very large number of processes executing concurrently, those processes have idle stages, and you want a simpler model to program with than other kinds of async.

Re: Java Virtual Threads: A Case Study

#150

Post by kbolino » Wed, Jul 17, 2024, 10:01 PM UTC

exabrial wrote: post #2

What is the virtual thread / event loop pattern seeking to optimize? Is it context switching? A number of years ago I remember trying to have a sane discussion about “non blocking” and I remember saying “something” will block eventually no matter what… anything from the buffer being full on the NIC to your cpu being at anything less than 100%. Does it shake out to any real advantage?

The hardware now is just as concurrent/parallel as the software. High-end NVMe SSDs and server-grade NICs can do hundreds to thousands of things simultaneously. Even if one lane does get blocked, there are other lanes which are open.