Loom looks like it’s nicely solved the function coloring problem. This plus Graal makes me excited to pick up Clojure again.
Achieving 5M persistent connections with Project Loom virtual threads
51–60 of 150 posts
Re: Achieving 5M persistent connections with Project Loom virtual threads
#52Earlier quoted context omitted.
It is true that the experiment exercises the OS, but that's only part of the point. The other part is that it uses a simple, blocking, thread-per-request model with Java 1.0 networking APIs. So this is "achieving 5M persistent connections with (essentially) 26-year-old code that's fully debuggable and observable by the platform." This stresses both the OS and the Java runtime. So while you could achieve 5M in other w…
I think we're in agreement. Ignoring under the hood - Loom's programming paradigm (from the viewpoint of control flow) is the Threading programming paradigm. (Virtual)Thread-per-connection programming is easier and far more intuitive than asynchronous (i.e. callback-esque) programming. I still attest though - The 5M connections in this example is still a red herring. Can we get to 6M? Can we get to 10M? Is that a que…
I don't quite follow your argument.
Saying "Linux cannot handle 5M connections with one thread per connection" isn't a reasonable statement because no operating system can do that, they can't even get close. The resource usage of a kernel thread is defined by pretty fundamental limits in operating system architecture, namely, that the kernel doesn't know anything about the software using the thread. Any general purpose kernel will be unable to provision userspace with that many threads without consuming infeasible quantities of RAM.
The reason JVM virtual threads can do this is because the JVM has deep control and understanding of the stack and the heap (it compiled all the code). The reason Loom scalability gets worse if you call into native code is that then you're back to not controlling the stack.
Getting to 10M is therefore very much a question for the JVM as well as the operating system. It'll be heavily affected by GC performance with huge heaps, which luckily modern G1 excels at, it'll be affected by the performance of the JVM's userspace schedulers (ForkJoinPool etc), it'll be affected by the JVM's internal book-keeping logic and many other things. It stresses every level of the stack.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#53Earlier quoted context omitted.
It is true that the experiment exercises the OS, but that's only part of the point. The other part is that it uses a simple, blocking, thread-per-request model with Java 1.0 networking APIs. So this is "achieving 5M persistent connections with (essentially) 26-year-old code that's fully debuggable and observable by the platform." This stresses both the OS and the Java runtime. So while you could achieve 5M in other w…
This. Writing the sort of applications that I get involved with, it's frequently the case whilst it's true that 1 OS thread/java thread was a theoretical scalability limitation - in practice we were never likely to hit it (and there was always the 'get a bigger computer'). But: the complexity mavens inside our company and projects we rely upon get bitten by an obsessive need to chase 'scalability' /at all costs/. Whi…
A couple of points to consider.
1. Demanding scalability for inappropriate projects and at any cost is something I've seen too, and on investigation it was usually related to former battle scars. A software system that stops scaling at the wrong time can be horrific for the business. Some of them never recover, the canonical example being MySpace, but I've heard of other examples that were less public. In finance entire multi-year IT projects by huge teams have failed and had to be scrapped because they didn't scale to even current business needs, let alone future needs. Emergency projects to make something "scale" because new customers have been on-boarded, or business requirements changed, are the sort of thing nobody wants to get caught up in. Over time these people graduate into senior management where they become architects who react to those bad experiences by insisting on making scalability a checkbox to tick.
Of course there's also trying to make easy projects more challenging, resume-driven development etc too. It's not just that. But that's one way it can happen.
2. Rx type models aren't just about the cost of threads. An abstraction over a stream of events is useful in many contexts, for example, single-threaded GUIs.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#54A bit of a digression, but I’d love to see how much further one could go with a memory-optimized userland TCP stack, and storing the send and receive buffers on disk. A TCP connection state machine consists of a few variables to keep track of sequence numbers and congestion control parameters (no more than 100-200 bytes total), plus the space for send/receive buffers. A 4 TB SSD would fit ~125 million 16-KB buffer pa…
Presumably at 100M simultaneous connections the machine CPU would be saturated with setting up and closing them, without getting much actual work done. TCP connections seem too fragile to make it worth trying to keep them open for really long periods.
It's interesting to think about though, I agree. What are the next scaling bottlenecks now (for JVM compatible languages) threading is nearly solved?
There are some obvious ones. Others in the thread have pointed out network bandwidth. Some use cases don't need much bandwidth but do need intense routability of data between connections, like chat apps, and it seems ideal for those. Still, you're going to face other problems:
1. If that process is restarted for any reason that's a lot of clients that get disrupted. JVMs are quite good at hot-reloading code on the fly, so it's not inherently the case that this is problematic because you could make restarts very rare. But it's still a problem.
2. Your CPU may be sufficient for the steady state but on restart the clients will all try to reconnect at once. Adding jitter doesn't really solve the issue, as users will still have to wait. Handling 5M connections is great unless it takes a long time to reach that level of connectivity and you are depending on it.
3. TCP is rarely used alone now, it usually comes with SSL. Doing SSL handshakes is more expensive than setting up a TCP connection (probably!). Do you need to use something like QUIC instead? Or can you offload that to the NIC making this a non-issue? I don't know. BTW the Java SSL stack is written in Java itself so it's fully Loom compatible.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#55The experiment is about Java app, but the tweaks are at the O/S level. Does it mean any app (Java/not, Loom/not) can achieve target given correct tweak? Also, why are these not default for the O/S? What are we compromising by setting those values?
No, it doesn't. The reason the tweaks are at the OS level is because, apparently, Loom-enabled JVMs already scale up to that level without needing any tuning. But if you try that in C++ you're going to die very quickly.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#56I think a lot of people are missing the point. Go look at the sourcecode. Look at how simple it is - anyone who has created a thread with java knows what's happening. With only minor tweaks, this means your pre-existing code can take advantage of this with, basically, no effort. And it retains all the debuggability of traditional java thread (I.e: a stack trace that makes sense!) If you've spent any time at all deali…
Agreed it's simpler, but using NIO with one OS thread per core also has it's benefits. The context switch (how ever small) will cause latency when this solution is at saturation. I think they should write four tests: fiber, NIO and each with userspace networking (no kernel copying network memory) and compare them. Why Oracle is stalling removing the kernel for Java networking is surprising to me, they allready have a…
there's still a context switch with NIO, you're just doing it manually
Re: Achieving 5M persistent connections with Project Loom virtual threads
#57Loom is missing the point. Time has shown that bare threads are not a viable high-level API for managing concurrency. As it turns out, we humans don't think in terms of locks and condvars but "to do X, I first need to know Y". That maps perfectly onto futures(/promises). And once you have those, you don't need all the extra complexity and hacks that green threads (/"colourless async") bring in. I'd take a system that…
Threads have essentially the same API as Futures - normally you have some join of join handle and you can join a set of threads (the equivalent of awaiting a set of futures).
Threads don't require locks and condvars. You can use channels and scoped joins etc. if you want.
Give me some async code and I'll show you an easier threaded version.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#58Reminds of https://phoenixframework.org/blog/the-road-to-2-million-webs... Would love to see this extended to more Languages/Frameworks.
In theory once Graal adds support for it, any Graal/Truffle-compatible language can benefit.
IMHO it's only JVM+Graal that can bring this to other languages. Loom relies very heavily on some fairly unique aspects of the Java ecosystem (Go has these things too though). One is that lots of important bits of code are implemented in pure Java, like the IO and SSL stacks. Most languages rely heavily on FFI to C libraries. That's especially true of dynamic scripting languages but is also true of things like Rust. The Java world has more of a culture of writing their own implementations of things.
For the Loom approach to work you need:
a. Very tight and difficult integration between the compiler, threading subsystem and garbage collector.
b. The compiler/runtime to control all code being used. The moment you cross the FFI into code generated by another compiler (i.e. a native library) you have to pin the thread and the scalability degrades or is lost completely.
But! Graal has a trick up its sleeve. It can JIT compile lots of languages, and those languages can call into each other without a classical FFI. Instead the compiler sees both call site and destination site, and can inline them together to optimize as one. Moreover those languages include binary languages like LLVM bitcode and WASM. In turn that means that e.g. Python calling into a C extension can still work, because the C extension will be compiled to LLVM bitcode and then the JVM will take over from there. So there's one compiler for the entire process, even when mixing code from multiple languages. That's what Loom needs.
At least in theory. Perhaps pron will contradict me here because I have a feeling Loom also needs the invariant that there are no pointers into the stack. True for most languages but not once C gets involved. I don't know to what extent you could "fix" C programs at the compiler level to respect that invariant, even if you have LLVM bitcode. But at least the one-compiler aspect is not getting in the way.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#59Earlier quoted context omitted.
I think you're mixing specific synchronisation/communication mechanisms with the basic concept of a thread, which is simply the sequential composition of instructions that is known and observable by the runtime . If you like the future/promise API, that will work even better with threads, because then the sequence is a reified concept known to the runtime and all its tools. You'll be able to step through the sequence…
You can implement futures by just running each future as a thread, but it doesn't really give you much. It's a lot more complex to write a preemptive thread scheduler + delegating future scheduler than to just write a future scheduler in the first place. Especially when that future scheduler already exists and works, and the preemptive one is a multi-year research project away.
It gives you a lot (aside from the ability to use existing libraries and APIs): observability and debuggabillity.
Supporting tooling has been one of the most important aspects of this project, because even those who were willing to write asynchronous code, and even the few who actually enjoyed it, constantly complained — and rightly so — that they cannot easily observe, debug and profile such programs. When it comes to "serious" applications, observability is one of the most important aspects and requirements of a system.
Instead of introducing new kind of sequenatial code unit through all layers of tooling — which would have been a huge project anyway, we abstracted the existing thread concept.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#60Earlier quoted context omitted.
This. Writing the sort of applications that I get involved with, it's frequently the case whilst it's true that 1 OS thread/java thread was a theoretical scalability limitation - in practice we were never likely to hit it (and there was always the 'get a bigger computer'). But: the complexity mavens inside our company and projects we rely upon get bitten by an obsessive need to chase 'scalability' /at all costs/. Whi…
A couple of points to consider. 1. Demanding scalability for inappropriate projects and at any cost is something I've seen too, and on investigation it was usually related to former battle scars. A software system that stops scaling at the wrong time can be horrific for the business. Some of them never recover, the canonical example being MySpace, but I've heard of other examples that were less public. In finance ent…
One additional - as noted, it's been 26 years since Java's founding. Project Loom has been around since at least 2018 and still has no release date. It'll be cool for Java projects whenever it comes out, but I just...have a hard time caring right now. I can't use it for old codebases currently, and new codebases I'm not using one request per Java thread anyway (tbh - when it's my choice I'm not choosing the JVM at all). The space has moved, and continues to move. In no way to say the JVM shouldn't be adopting the good ideas that come along the way, that is one of the benefits of being as conservative and glacial in adoption as it is, but I just...don't get excited about them, or find myself in any position in relation to the JVM (Java specifically, but the fundamentals affect other languages) other than "ugh, this again".