API for the server example looks... actually good, wow. Nice job!
Also tickled to see my erlang 1M comet blog post referenced. A lifetime ago now, pre-websockets.
Achieving 5M persistent connections with Project Loom virtual threads
101–110 of 150 posts
Re: Achieving 5M persistent connections with Project Loom virtual threads
#102A bit of a digression, but I’d love to see how much further one could go with a memory-optimized userland TCP stack, and storing the send and receive buffers on disk. A TCP connection state machine consists of a few variables to keep track of sequence numbers and congestion control parameters (no more than 100-200 bytes total), plus the space for send/receive buffers. A 4 TB SSD would fit ~125 million 16-KB buffer pa…
Presumably at 100M simultaneous connections the machine CPU would be saturated with setting up and closing them, without getting much actual work done. TCP connections seem too fragile to make it worth trying to keep them open for really long periods. It's interesting to think about though, I agree. What are the next scaling bottlenecks now (for JVM compatible languages) threading is nearly solved? There are some obv…
You're totally spot on that connection establishment is much more challenging than steady state; with TLS or just TCP.
I don't think QUIC helps with that at all. Afaik, QUIC is all userland, so you'd skip kernel processing, but that doesn't really make establishment cheaper. And TCP+TLS establishes the connection before doing crypto, so that saves effort on spoofing (otoh, it increases the round trips, so pick your tradeoffs).
One nice thing about TCP though is it's trivial to determine if packets are establishing or connected; you can easily drop incoming SYNs when CPU is saturated to put back pressure on clients. That will work enough when crypto setup is the issue as well. Operating systems will essentially do this for you if you get behind on accepting on your listen sockets. (Edit) syncookies help somewhat if your system gets overwelmed and can't keep state for all of them half-established connections, although not without tradeoffs.
In the before times, accelerator cards for TLS handshakes were common (or at least available), but I think current NIC acceleration is mainly the bulk ciphering which IMHO is more useful for sending files than sending small data that I'd expect in a large connection count machine. With file sending, having the CPU do bulk ciphers is a RAM bottleneck: the CPU needs to read the data, cipher it, and write to RAM then tell the NIC to send it; if the NIC can do the bulk cipher that's a read and write omitted. If it's chat data, the CPU probably was already processing it, so a few cycles with AES instructions to cipher it before sending it to send buffers is not very expensive.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#103Earlier quoted context omitted.
There have been userspace thread libraries for c++ for decades.
Sure, I wrote some myself. Q is what libraries you can use on top of the userspace thread package that are aware of the userspace threads rather than just using OS APIs and thus eg blocking the current OS thread.
There are .so interposition tricks that can be used for that.
I think Pth used to do that for example.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#104Earlier quoted context omitted.
https://github.com/ebarlas/project-loom-comparison
Shouldn’t you be able to send authorization and authentication requests in parallel in the async and virtual threads cases?
It is just an example so they could do anything.
But in the real world it is common to need information from the authorization stage to use in the authentication stage. For example you may have a user login with an email address/password which you then pass to an LDAP server in order to get a userId. This userId is then used in a database to determine with objects/groups they have access to.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#105This is pretty fantastic! I'm very excited about the possibilities of Loom. Would love to have a more realistic sample with Spring Boot that would demonstrate the real world scale. I saw a few but nothing remotely as ambitious as that.
Spring Boot overhead would likely make that infeasible.
Spring boot overhead is largely in startup time. It really doesn't have much overhead there after.
It's largely a collection of the same libraries you would use anyways glued together with a custom di system.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#106I see a lot of these making the FP of HN. But it's very difficult to be impressed, or unimpressed because it's all about hardware. How much hardware is everybody throwing at all of this? 5M persistent connections on a Pi with mere GigE? Pretty frickin' amazing. 5M persistent connections on a Threadripper with 128 cores and a dozen trunked 4 port 10GE NICs? Yaaaaawwwnnn snooze.
We need a standardized computer for benchmarking these types of claims. I propose the RasPi 4 4GB model. Everybody can find one, all the hardware's soldered on so no cheating is really possible, etc. Then we can really shoot for efficiency.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#107I think a lot of people are missing the point. Go look at the sourcecode. Look at how simple it is - anyone who has created a thread with java knows what's happening. With only minor tweaks, this means your pre-existing code can take advantage of this with, basically, no effort. And it retains all the debuggability of traditional java thread (I.e: a stack trace that makes sense!) If you've spent any time at all deali…
Or inserting the occasional Task.Run() calls, as means to avoiding changing the whole call stack up to Main().
This hasn't been that much of a problem, IME
If you decide somewhere deep in your program you want to use async operations, most languages allow you to keep the invoking function/closure synchronous and return some kind of Promise/Future-like value
Re: Achieving 5M persistent connections with Project Loom virtual threads
#108I was only able to get to 840,000 open connections with my experiment. My machine only has 8GB of memory. https://josephmate.github.io/2022-04-14-max-connections/ Is there anyway for the TCP connections share memory in kernel space? My experiment only uses two 8 byte buffers in userspace.
Does Linux actually allocate buffers for each socket or does it just link to sk_buff's (which I understand are similar to FreeBSD's mbuf's) and then limit how much storage can be linked? FreeBSD has a limit on the total ram used for mbufs as well, not sure about Linux.
Otoh, FreeBSD's maximum FD limit is set as a factor of total memory pages (edit: looked it up, it's in sys/kern/subr_param.c, the limit is one FD per four pages, unless you edit kernel source) and you've got 2M pages with 8GB ram, so you would be limited to 512k FDs total, and if you're running the client on the same machine as server, that's 256k connections. But 8G is not much for a server, and some phones have more than that... so it's not super limiting.
When you're really not doing much with the connections, userland tcp as suggest in a sibling, could help you squeeze in more connections, but if you're going to actually do work, you probably need more ram.
Btw, as a former WhatsApp server engineer, WhatsApp listens on three ports; 80, 443, and 5222. Not that that makes a significant difference in the content.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#109I see a lot of these making the FP of HN. But it's very difficult to be impressed, or unimpressed because it's all about hardware. How much hardware is everybody throwing at all of this? 5M persistent connections on a Pi with mere GigE? Pretty frickin' amazing. 5M persistent connections on a Threadripper with 128 cores and a dozen trunked 4 port 10GE NICs? Yaaaaawwwnnn snooze. We need a standardized computer for benc…
Raspberry pi 4 performance changes wildly based on cooling. Bare die vs heatsink vs heatsink + fan will give you wildly different results.
Re: Achieving 5M persistent connections with Project Loom virtual threads
#110I see a lot of these making the FP of HN. But it's very difficult to be impressed, or unimpressed because it's all about hardware. How much hardware is everybody throwing at all of this? 5M persistent connections on a Pi with mere GigE? Pretty frickin' amazing. 5M persistent connections on a Threadripper with 128 cores and a dozen trunked 4 port 10GE NICs? Yaaaaawwwnnn snooze. We need a standardized computer for benc…
Raspberry pi 4 performance changes wildly based on cooling. Bare die vs heatsink vs heatsink + fan will give you wildly different results.
Same is true with any computer these days. So let's go no heat sink, Pi 4 4GB anyway.