Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

blog.ydb.tech

Post reply on HN
Print view View original

241–250 of 253 posts

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#241

Post by dikei »

skyde wrote: post #234

Why synchronized block are not preemptible? When compiling public void syncMethod() { synchronized(lock) { // some code } } they could translate to public void syncMethod() { await reentrantLockAsync.lockAsync(); try { await somecodeAsync(); } finally { await lock.unlockAsync() } }

The first issue is your second code is not Java (no await/async literal for Java yet)

The second issue is they're not completely equivalent. In the second case, you'd need extra memory for the `reentrantLock`, while `synchronized` works with any object. Furthermore, if you need to use `wait/notify`, then there need to be an extra `Condition` object to use in combination with the `ReentrantLock`. For sure, developers can rewrite most `synchronized` to use `ReentrantLock` and `Condition`, but javac won't do it automatically for you.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#242

Post by dxxvi »

Why was c3p0 used (its latest version was released in Dec 2019)? Those tests existed for a while and people were too lazy to replace c3p0 with something newer? I guess that they spent all their time to use virtual threads in those tests and had no time left to look at c3p0.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#243

Post by josefx »

bunderbunder wrote: post #238
merb wrote: post #230

Earlier quoted context omitted.

In .net async won where lock and mutex does not work (lock is like synchronized, not exactly the same, tough). That’s why most libraries use SemaphoreSlim which would work with green threads. But that’s more because of the ecosystem. I’ve barley stumble upon lock’s and mutex is mostly used in the main method since it acquires a real os mutex, not really a cheap thing but for GUIs it’s clever to check if the app is ru…

Yeah, definitely. But for a fair comparison I think you have to look at how .NET did things before async/await hit the scene. And, for that, the aspect of the design in question is quite similar between the two.

Early .Net is hardly an independent data point from early Java. Not only was .Net directly influenced by Java, it also had to support a direct migration from the Microsoft JVM specific Visual J++ to J#.

The handful of languages I know either do not have a top level object class that supports a randomized set of features ( C++ ) or prioritize a completely different way of concurrent execution ( Python, JavaScript ).

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#244

Post by Groxx »

xmcqdpt2 wrote: post #239
Groxx wrote: post #226

Earlier quoted context omitted.

The only way I can see this being a problem is if the virtual threads can't be stolen from their (now pinned) carrier thread. Because otherwise that's all true of real threads too, blocking them is the whole point of Object.wait. If there's no work-stealing from pinned carriers (or they're low-finite and normal threads are effectively infinite): yes that'd be a HUGE issue. I would be shocked if they released anything…

> If there's no work-stealing from pinned carriers (or they're low-finite and normal threads are effectively infinite): yes that'd be a HUGE issue. I would be shocked if they released anything with that limitation though, that would violate some of the core expectations of mutexes and threads - independent ones need to make progress or nearly all patterns can't guarantee progress. Correct you can't steal the carrier…

>Correct you can't steal the carrier thread from an Object.wait() waiting virtual thread. This is apparently in the pipeline but it is a pretty major limitation.

I mean stealing other virtual threads from the pinned carrier thread (except for the one pinning it) so they can make progress. Normal work-stealing stuff - the queue(thread) is blocked(pinned), so process that task(virtual thread) in a different queue(thread).

It makes sense that a pinned thread remains pinned with the virtual thread that pinned it.

The 256 default carrier thread limit is going to frequently be a problem though, yeah. That's more than enough to cause all this, and it's a pretty crazy default imo.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#245

Post by simiones »

incrudible wrote: post #240
bitzun wrote: post #223

Earlier quoted context omitted.

Most application webservers (by default) handle one request per thread. For mostly IO bound stuff (which many projects are), it makes sense to me that threads become a bottleneck in relatively ordinary scenarios.

The scenario where your IO could handle way more than a thousand concurrent requests if only the thread overhead was reduced? When does that ever happen?

Each OS thread costs memory. With the version of Java I have, the default is to allocate 1MB of stack for each thread. So, 10,000 threads would require 10,000 MB of RAM even if we configured ulimit to allow that many threads. In contrast, asking the kernel to do buffered reads of 10,000 files in parallel requires much less memory - especially if most of those are actually the same physical file. Of course, they won't be read fully in parallel.

For example, this program:

  var threads = new Thread[20000];
  for (int i = 0; i  {
      try {
        Files.copy(FileSystems.getDefault().getPath("abc.txt"), System.out);
      } catch (IOException e) {
        System.err.println("Error writing file");
        e.printStackTrace();
      }});
    }
  for (int i = 0; i
Run as `java Test > ./cde.txt` takes about 4.5s to run on my WSL2 system with 2 cores, writing a 2 GB file (with abc.txt having 100KB); even this would be within the HTTP timeout, though users would certainly not be happy. Pretty sure a native Linux system on a machine beefy enough to be used as a web server would have no problem serving even larger files over a network like this.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#246

Post by pjmlp »

coldtea wrote: post #192
simiones wrote: post #176

Earlier quoted context omitted.

> Virtual threads are nice for unblocking legacy code but they aren't without issues. There are better options for new code with less trade offs on the jvm as well. The designers of Project Loom would say the exact opposite. The whole push behind Project Loom and similar models (Go's oft-praised "goroutines" runtimes being another one) is motivated by Threads being a much better fit for async behavior in a fundamenta…

> The designers of Project Loom would say the exact opposite. Sure, but then again the designers of circa 2000-2010 J2EE also thought the verbosity and over-engineering was a good idea.

J2EE started as a Objective-C framework, before being rewritten in Java.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#247

Post by tveita »

papercrane wrote: post #17

Curious if you considered switching to a different connection pooling library. These days I usually use HikariCP which is fast an actively maintained. c3p0 hasn't had any activity for years, I'm not sure if it's still maintained.

Looks like HikariCP is also awaiting fixes for this https://github.com/brettwooldridge/HikariCP/pull/2055

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#248

Post by incrudible »

simiones wrote: post #245
incrudible wrote: post #240

Earlier quoted context omitted.

The scenario where your IO could handle way more than a thousand concurrent requests if only the thread overhead was reduced? When does that ever happen?

Each OS thread costs memory. With the version of Java I have, the default is to allocate 1MB of stack for each thread. So, 10,000 threads would require 10,000 MB of RAM even if we configured ulimit to allow that many threads. In contrast, asking the kernel to do buffered reads of 10,000 files in parallel requires much less memory - especially if most of those are actually the same physical file. Of course, they won't…

1. You are not solving a real problem. The use case you describe (basically a CDN) is already exotic, the scenario where such a system would have already been implemented with Java and its basic IO seems implausible.

2. You did not compare against fewer threads to see if threads are actually the bottleneck rather than IO. Also, all your threads are competing for stdout.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#249

Post by marwis »

pjmlp wrote: post #90
dboreham wrote: post #63

Earlier quoted context omitted.

My long held belief: green/user-level/M:N threading schemes never work at first, and only work reliably after extreme effort has been put into fixing all the cases where blocking code gets called underneath. afaik there are only two modern working implementations: golang and erlang. This article is consistent with that belief.

Go suffers the same issue when calling into native code, that is why it has APIs to deal with it. For example, https://pkg.go.dev/runtime#LockOSThread

This seems different.

It pins goroutine until it is explicitly released ensuring that multiple native calls will remain on the same platform thread and nothing else is going to use it. This is critical for namespace manipulation on Linux.

Java only pins for duration of native call and synchronized blocks.

It looks like Java does not offer equivalent API? For now could be achieved with synchronized but if synchronized will be changed in the future to not pin it would break.

Re: We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres

#250

Post by marwis »

marwis wrote: post #249
pjmlp wrote: post #90

Earlier quoted context omitted.

Go suffers the same issue when calling into native code, that is why it has APIs to deal with it. For example, https://pkg.go.dev/runtime#LockOSThread

This seems different. It pins goroutine until it is explicitly released ensuring that multiple native calls will remain on the same platform thread and nothing else is going to use it. This is critical for namespace manipulation on Linux. Java only pins for duration of native call and synchronized blocks. It looks like Java does not offer equivalent API? For now could be achieved with synchronized but if synchronized…

Oh, actually one can just spawn non-virtual thread to solve it.
Post reply on HN