Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

blog.ydb.tech

Post reply on HN
Print view View original

191–200 of 253 posts

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#191

Post by xmcqdpt2 »

eivanov89 wrote: post #30
papercrane wrote: post #17

Curious if you considered switching to a different connection pooling library. These days I usually use HikariCP which is fast an actively maintained. c3p0 hasn't had any activity for years, I'm not sure if it's still maintained.

Perhaps we'll give HikariCP a chance. However, please keep in mind that the goal of the YDB team is to enhance database performance. We needed virtual threads to make TPC-C efficient enough to generate a reasonable load on a modest amount of hardware.

Is that true?

Virtual threads aren't necessarily faster, you still have just as many sockets and network connections as before. You can easily spawn 5000 platform threads, and if that's not enough, there are quite a few user-space implementations of fibers/coroutines/async etc on the JVM that can deal with many outlying requests (Cats/ZIO in Scala, Kotlin coroutines, the Play framework, concurrent.Future, etc.)

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#192

Post by coldtea »

simiones wrote: post #176
jillesvangurp wrote: post #115

Earlier quoted context omitted.

People always forget that things that only happen every few million times, can happen fairly frequently on a busy server. This has bitten me numerous times. The nature of a lot of these types of issues is that they are hard to detect and hard to reproduce. Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well.…

> Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well. The designers of Project Loom would say the exact opposite. The whole push behind Project Loom and similar models (Go's oft-praised "goroutines" runtimes being another one) is motivated by Threads being a much better fit for async behavior in a fundamenta…

>The designers of Project Loom would say the exact opposite.

Sure, but then again the designers of circa 2000-2010 J2EE also thought the verbosity and over-engineering was a good idea.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#193

Post by xmcqdpt2 »

delusional wrote: post #80
avodonosov wrote: post #55

Earlier quoted context omitted.

Java does the same for Object.wait(), only the number of such compensating threads is limited by default, but can be extended via config option. They have exhausted the default number of compensating threads, I think. And they are mistaken to call this situation a "pinning" JEP 444: > The vast majority of blocking operations in the JDK will unmount the virtual thread, freeing its carrier and the underlying OS thread…

> The situation with Object.wait() is not what JEP 444 calls "pinning". The "pinning" happens, for example, when one calls `syncronized(....) {blockingQueue.take()}` [...] To call Object.wait() you need to own the objects monitor, which would imply that your code would actually look like `synchronized(....) {Object.wait()}` in which case you would indeed be pinned.

Object.wait() releases the monitor lock though. This specific case doesn't have to do with synchronized at all, but with wait() being a native call.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#194

Post by diggan »

logifail wrote: post #123
iknowstuff wrote: post #62

Earlier quoted context omitted.

They do add value, they make clicks more likely.

> They do add value, they make clicks more likely "Making clicks more likely" is a terrible measure of genuine value. There are lots of images which will make people click, even if once they see your page they click 'Back' a second later. Our metrics are broken if we continue to attribute that click as 'success'.

> "Making clicks more likely" is a terrible measure of genuine value.

Genuine value, to who? For the author, getting more clicks is probably of "genuine value", depending on their goals for their writing. But seems most people are not writing and publishing stuff today because they think it provides value to others, but because they think it'll provide value to themselves somehow.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#195

Post by jillesvangurp »

simiones wrote: post #176
jillesvangurp wrote: post #115

Earlier quoted context omitted.

People always forget that things that only happen every few million times, can happen fairly frequently on a busy server. This has bitten me numerous times. The nature of a lot of these types of issues is that they are hard to detect and hard to reproduce. Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well.…

> Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well. The designers of Project Loom would say the exact opposite. The whole push behind Project Loom and similar models (Go's oft-praised "goroutines" runtimes being another one) is motivated by Threads being a much better fit for async behavior in a fundamenta…

I think it's not that black and white. Clearly they made a choice to be backwards compatible. Not because Java Threads have a nice API (not even close) but because a lot of legacy code that will never be changed uses it. Including all the ugly bits that you shouldn't be using. Like a lot of the low level synchronization primitives that date back to the early days of Java. It's an impressive bit of work but they made some compromises to make things work. A new API would have been easier, would have had less overhead, and be nicer to use. But backwards compatibility with legacy code was a big goal.

It mostly works fine and it's an impressive bit of engineering. But it has some really ugly failure modes in combination with hacky legacy code designed for real threads. So, you can't blindly assume things to just work. Hence the deadlocks.

Many Java servers already work the way you outline. It's just that they are a bit tedious to use with the traditional Java frameworks. Which is one reason I like using Spring's webflux with Kotlin instead. Just way nicer when it's all exposed via co-routines.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#196

Post by bheadmaster »

dikei wrote: post #149
davidtos wrote: post #143

Earlier quoted context omitted.

That already exists luckily, you can even change the maximum number of carrier threads with: - jdk.virtualThreadScheduler.maxPoolSize=10

The default is 256, way higher than 10. But of course, when you have thousands of Virtual Threads all deliberately pinning the carrier thread, you quickly run out.

I suppose that a hard limit on the number of carrier threads is a sensible choice then - deadlock is better than creating threads until the system grinds to a halt.

But then again, why couldn't scheduler detect a deadlock? Go has a system in place that, in case of total program deadlock, prints out an error message with all goroutines' stack traces, and stops the program. Perhaps Virtual Thread Scheduler could do the same thing?

But then again, Java also allows for native threads to run in parallel to Virtual Threads, which makes it impossible to detect whether there's a deadlock, and not just virtual threads waiting on a native thread.

I suppose this is a very good example why simple is better than complex.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#197

Post by layer8 »

blindriver wrote: post #66
mrintegrity wrote: post #56

Totally off topic but I am getting tired of the AI generated images used on nearly all blog posts nowadays. They are instantly recognisable, it just seems low effort and lowers the feeling of quality one might otherwise have

I prefer AI generated images over stock photos though. You can tell that both are phony, but at least the AI can be a bit more creative.

Th issue is that we’re now getting tons of blog posts with AI-generated images that previously didn’t dare to use stock photos.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#198

Post by discreteevent »

coldtea wrote: post #192
simiones wrote: post #176

Earlier quoted context omitted.

> Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well. The designers of Project Loom would say the exact opposite. The whole push behind Project Loom and similar models (Go's oft-praised "goroutines" runtimes being another one) is motivated by Threads being a much better fit for async behavior in a fundamenta…

> The designers of Project Loom would say the exact opposite. Sure, but then again the designers of circa 2000-2010 J2EE also thought the verbosity and over-engineering was a good idea.

There might be some justification for comparing any one particular thing to the worst possible particular thing if those things have something in common. The only feature the two things you picked have in common is the word 'java'.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#199

Post by deely3 »

i000 wrote: post #187
pron wrote: post #172

Earlier quoted context omitted.

As the JEP states, pinning due to synchronized is a temporary issue. We didn't want to hold off releasing virtual threads until that matter is resolved (because users can resolve it themselves with additional work), but a fix already exists in the Loom repository, EA builds will be offered shortly for testing, and it will be delivered in a GA release soon. Those who run into this issue and are unable or unwilling to…

Wow, I would love to be in the meeting where this decision was made. Let's ship this with a foot gun, but lets not mention in the JEP that it may hang - let them figure it out.

I understand the frustration, but why not read a doc?

https://docs.oracle.com/en/java/javase/21/core/virtual-threa...

In Virtual Threads: An Adoption Guide part there is:

When using virtual threads, if you want to limit the concurrency of accessing some service, you should use a construct designed specifically for that purpose: the Semaphore class.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#200

Post by bilbo0s »

i000 wrote: post #187
pron wrote: post #172

Earlier quoted context omitted.

As the JEP states, pinning due to synchronized is a temporary issue. We didn't want to hold off releasing virtual threads until that matter is resolved (because users can resolve it themselves with additional work), but a fix already exists in the Loom repository, EA builds will be offered shortly for testing, and it will be delivered in a GA release soon. Those who run into this issue and are unable or unwilling to…

Wow, I would love to be in the meeting where this decision was made. Let's ship this with a foot gun, but lets not mention in the JEP that it may hang - let them figure it out.

I don't know man?

We make scalable graphics rendering servers to stream things like videogames across the web. When we started the project to switch to virtual threads we had that as number one on the big board. "Rewrite for reentrant locks."

Maybe we have more fastidious engineers than a normal company would since we are in the medical space? But even the juniors were reading and familiarizing themselves on how to properly lock in loom's infancy.

All that only to point out that, yes, they had communicated the proper use of reentrant locks long ago.

I do understand what you're saying from an engineering management perspective though. That effort cost a fortune. Especially when you have the FDA to deal with.

It was more than worth it though! In the world of cloud providers, efficiency is money.

Post reply on HN