Live data from Hacker News

We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

blog.ydb.tech

71–80 of 253 posts

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#71
post #10

It is a known caveat that virtual threads do not work well with long running synchronization by pinning the thread. That unfortunately means that for many applications it may be premature to adopt them, but it is mature enough for broader evaluation by the libraries and frameworks. The Java team provided a status of their efforts recently [1]. https://www.youtube.com/watch?v=WoQJnnMIlFY&t=421s

Sorry, the first sentence is a mis-informing wording.

The `synchronized` pins the thread only when from within of the `synchronized` the program calls a blocking operation that would normally unmount the virtual thread, like blockingQueue.take() or similar. (Which is not a sane coding practice). It's because the unmounting, as it's implemented today, does not work well with synchronized.

It's better if people read JEP 444 than rely on forum comments, to avoid being misinformed.

Speaking of long-running - even without synchronized, a long running code keeps the native thread occupied, until some blocking operation is called. So an endless loop that does not call a virtual-thread-ready blocking operation will occupy the native thread forever.

Java virtual threads are a kind of cooperative multithreading - another virtual thread only gets chance to kick-in when some current virtual thread reaches specific blocking operations. In contrast to preemptive multi-threading with native threads.

So I agree with your conclusion. Virtual threads can not (yet?) be blindly used as a drop-in replacement of native threads for existing code. And the new code needs to take their specifics into account.

BTW, another method I discovered to block the native carrier thread that executes a virtual thread is to call blocking reading through FileInputStream, for example reading from the console. The FileInputStream does not implement virtual thread parking at all (yet?).

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#72
This problem is not going to go away so easily. Numerous core Java classes (like BufferedInputStream) use synchronized. I count 1600+ usages in java.base. The blocking issue means it's _much_ easier to accidentally run into this, rather than waving it away as an unlikely edge case.

I personally ran into this Using the built in com.sun webserver, with a virtual thread executor. My VPS only has two CPUs which means the FJP that virtual threads run on only have 2 active threads at a time. I ran into this hang when some of the connection hung, blocking any further requests from being processed.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#73
post #10

It is a known caveat that virtual threads do not work well with long running synchronization by pinning the thread. That unfortunately means that for many applications it may be premature to adopt them, but it is mature enough for broader evaluation by the libraries and frameworks. The Java team provided a status of their efforts recently [1]. https://www.youtube.com/watch?v=WoQJnnMIlFY&t=421s

Sorry, the first sentence is a mis-informing wording. The `synchronized` pins the thread only when from within of the `synchronized` the program calls a blocking operation that would normally unmount the virtual thread, like blockingQueue.take() or similar. (Which is not a sane coding practice). It's because the unmounting, as it's implemented today, does not work well with synchronized. It's better if people read JE…

By long running I just meant anything that was not fast compute. I was more focused on finding the reference link so I agree my wording wasn’t clear.

Go started without preemption and added it later. The Java team has indicated a similar path, so we might see that tackled in the future. I think they could do that using safe points or JEP 312‘s handshakes, so it’s not infeasible.

For file io they wanted to explore io_ring and they might need to add a loom friendly resolver for JEP 418. There is just so much left, like scalable timers, that I think it’s going to be a long time until VTs will be a good default choice.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#74

Totally off topic but I am getting tired of the AI generated images used on nearly all blog posts nowadays. They are instantly recognisable, it just seems low effort and lowers the feeling of quality one might otherwise have

Worse yet, the dining philosophers in the image have too many hands. No wonder they’re deadlocking! :)

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#75
post #12

Earlier quoted context omitted.

I would like to reply with a quote of sir Tony Hoare's 1980 ACM Turing Award Lecture: "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies".

This one was a very large caveat pointed out loudly every time the feature was mentioned to the community. So, yes the limitation/flaw was very well known and is ideally going to be addressed in a future JDK release.

> This one was a very large caveat pointed out loudly every time the feature was mentioned to the community.

It really wasn't. There were people on here, including Oracle employees, claiming that the virtual thread implementation was a drop-in replacement that would work (not necessarily perform better, but work) in all cases.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#76
post #34

Earlier quoted context omitted.

The issue here is currently virtual threads don't work well with the 'sychronized' keyword. Right now synchronized will pin the carrier thread. The fix was to switch to a higher-level abstraction that works with virtual threads. My understanding is there is work to make synchronized not pin the carrier thread, but that's some pretty complex and important code to change.

From a relatively brief skim and past Go and Java experience: synchronized blocks the current normal thread, so that doesn't really seem any different to me. If you starve your threads, you starve your threads. It definitely leaves room to optimize by not pinning that thread, which would be great, but that shouldn't change semantics at all. Or is there something actually screwed up in the implementation of virtual th…

Java's virtual threads are supposed to be a drop-in replacement for real threads. But using virtual threads means you get a far smaller number of real threads, and things that were safe back when you had an unlimited number of real threads available (or at least, a larger number than your database connection pool) are no longer safe.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#77

Curious if you considered switching to a different connection pooling library. These days I usually use HikariCP which is fast an actively maintained. c3p0 hasn't had any activity for years, I'm not sure if it's still maintained.

Indeed, Hikari is the go to connection pool for some years now. It's even the default when running Spring Boot.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#78

Totally off topic but I am getting tired of the AI generated images used on nearly all blog posts nowadays. They are instantly recognisable, it just seems low effort and lowers the feeling of quality one might otherwise have

Typically it’ll be in lieu of nothing or stock photography. Doesn’t it seem better than that?

One is aesthetic filler that is true to its purpose of loosening up the typography of a wall of text. The other tends to be awkwardly clever on a level of awkward that was unknown to mankind until recently. I used to hate stock photography fillers just like everybody else, but now my preference is as clear as it would be surprising to past me.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#79

Totally off topic but I am getting tired of the AI generated images used on nearly all blog posts nowadays. They are instantly recognisable, it just seems low effort and lowers the feeling of quality one might otherwise have

I prefer AI generated images over stock photos though. You can tell that both are phony, but at least the AI can be a bit more creative.

I prefer no images over IA stolen/generated images.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#80
post #45

Go has a mechanism to spawn a new thread (m in ho runtime parlance) if it thinks one of its threads might be blocked in a cgo (go’s “native function” equivalent). That prevents stuff like this.

Java does the same for Object.wait(), only the number of such compensating threads is limited by default, but can be extended via config option. They have exhausted the default number of compensating threads, I think. And they are mistaken to call this situation a "pinning" JEP 444: > The vast majority of blocking operations in the JDK will unmount the virtual thread, freeing its carrier and the underlying OS thread…

> The situation with Object.wait() is not what JEP 444 calls "pinning". The "pinning" happens, for example, when one calls `syncronized(....) {blockingQueue.take()}` [...]

To call Object.wait() you need to own the objects monitor, which would imply that your code would actually look like `synchronized(....) {Object.wait()}` in which case you would indeed be pinned.

Post reply on HN