Every Byte Matters

>HotSpot uses much more RAM by design as there are inefficiencies caused by using too little of it.

Ah yes, the swapping induced by IntelliJ overflowing my system RAM is supposed to reduce the inefficiencies of using too little memory. Great...

Thanks pron, you've fully bought into all the JVM kool-aid talking points without ever trying to question them. One of the reasons I upgraded to 32 GB RAM in 2019 was to run a Minecraft modpack. Minecraft is one of the most memory intensive games I've ever played.

When you consider that the smallest cloud instances that cost $4 per month only give you like 512 MB of RAM and have refused to upgrade for at least a decade, the idea of using more than 512 MB to be "more efficient" is ridiculous. It raises your minimum costs to $10 per month.

>I gave a talk on the subject that I hope will be published soon, and while I can't reproduce it here, let me give an example that offers some basic intuition.

>Imagine needing to do some computation in two ways on a machine with 1GB of free RAM. You could run for 10s, taking up 100% CPU and consuming 80MB of RAM, or for 9s, taking up 100% CPU and consuming 800MB of RAM.

This is the "wasted RAM is unused RAM" mentality and it doesn't work, because you usually have multiple competing programs and when you run out of RAM, your system will start swapping. This will then require you to buy more RAM, leading to more leftover RAM, which is then wasted and gets consumed by the applications again. It's nonsense.

Then there is the fact that the vast majority, basically 99.9% of algorithms are not scalable in the naive way presented. Nobody will waste resources on writing the same algorithm twice for these two cases. Databases are usually designed to either be primarily file system backed or in-memory backed. They will use the extra memory to hold indices and let the OS do the caching or they will reserve all the memory up front, intentionally leaving nothing for other applications.

>The second is more efficient, despite taking up 10x more RAM and saving "only" 10% of CPU, regardless of the relative cost of RAM and CPU. This is because taking up 100% of the CPU effectively captures 100% of RAM (as no other program can use it), so both programs capture the entire 1GB only the second one captures it for a second less.

Ok, now you're just writing nonsense. Nowadays people have CPUs with multiple cores and use an OS with a scheduler. If you have two programs taking up 100% of the CPU, the OS will give each process some of the hardware resources. You can't just assume some 100% CPU blockage here just because it is convenient for your argument. It's especially dishonest since even a 99% CPU blockage basically makes your argument fall apart completely.

If you have two programs decide to 10x the memory consumption to save one second, you'll most likely run into swapping issues, which will actually lock up your system for several seconds at a time and if you're unlucky, the OOM killer strikes or the compositor freezes up and you have to reboot. You're saying that a 1 second savings is worth an endless amount of inconveniences.

>This scales to non extreme situations because accessing RAM requires CPU, so using CPU means capturing RAM whether you use it or not. So HotSpot uses it if it can use it to balance the CPU utilisation.

Again, this is completely incorrect in so many ways that you're bragging you know nothing about how modern computers work.

CPU cores have their own local memory resources called caches. Depending on how your code is written, you may tile your data so it fits entirely in cache and operate within the local memory.

When performing inter thread communication, there are often situations where the data often doesn't even get written and then loaded to main memory, since atomic operations can make use of the MESI cache coherency protocol to pull the data directly from another cores' cache.

Nowadays DMA is the standard way to perform large data transfers to hardware peripherals. If you load a file from an HDD, the SATA peripheral will communicate via DMA to copy whole sectors or file system blocks. The same applies to sending data to an SSD, network interface, GPU or basically anything else that performs bulk transfers (1 KiB+). The DMA engine is a separate component independent of the CPU and it may write data directly into cache as well.

Then there is the fact that RAM is a form of storage and storage is usually characterized by the fact that it takes up an area and said areas can be subdivided. When RAM is used, the portion of used RAM is considered blocked for the duration of how long it is stored, independently of whether it is accessed or not. This means that the most important objective is having sufficient amounts of RAM to store all data, not to occupy all of it preemptively even when it is not really needed.

The same can't be said of CPUs. Occupying the CPU usually means actively using the CPU. The only exception to this is things like spinlocks which should be avoided like the plague. By what the CPU is occupied is determined by the OS, therefore your logic is backwards. It's not the program blocking the CPU and therefore blocking the memory. The OS decided to stop running your process to run another process. Progress is slowed down, but it is not blocked.

Actual blockage only occurs when two processes compete for a fixed resource so that it is not possible to run both processes simultaneously, so that one process has to be closed to run another process.

You're right, because I certainly don't consider resource-constrained programs to be the only performance-sensitive applications. I consider an application performance-sensitive when it has severe performance requirements (either on throughput or latency or both) that aren't easily or sufficiently met with horizontal scaling. This typically involves situations where high volumes of data must flow and be processed on the same machine (my own journey with Java began when we ported a large C++ application that did distributed, soft-realtime sensor fusion, synchronised with atomic clocks, to Java, and it was very much performance-sensitive).

If you are running in a resource-constrained environment, you might have no choice but to have complete control over hardware resources, in which case you may need to use a low-level language, but your optimisation budget is very high. A different and more common case is where the hardware isn't too resource-constrained, but the performance requirements aren't easily met, either. In these situations, the performance challenge isn't necessarily to optimise at all costs, but to find a way to meet the performance requirement while staying within budget. In these areas, Java has already displaced C++, and continues to be the first language of choice.

Of course, the people who write such applications (in any language) don't often talk about their architecture, but here's one example when they do: https://www.infoq.com/presentations/java-robot-swarms/ In this case, as in many others, the performance requirements are strict (and aren't easily met with horizontal scaling), but the constraint under which they must be met isn't the hardware but the budget and speed of development/evolution.

More often, the performance challenge is how to get the best performance per unit of effort (while meeting the performance requirements, of course) rather than how to get the last 1-5% of performance at any cost. Or sometimes I put this question as not "how fast can a program be?" but "how fast can I practically make my program?"

The optimisations Java offers are precisely intended to maximise the latter, because that's exactly where low-level languages suffer performance shortcomings. They could get that performance or perhaps better with a lot more effort (that needs to be continuously spent throughout the software's lifetime), but many performance-sensitive applications don't have or would rather not spend the time, money, or expertise to do that, and are looking for the best performance per unit of effort.

> Ah yes, the swapping induced by IntelliJ overflowing my system RAM is supposed to reduce the inefficiencies of using too little memory. Great...

That's like me saying, oh great, so the swapping introduced by MS Word or Outlook shows just how efficient C++ is...

> Thanks pron, you've fully bought into all the JVM kool-aid talking points without ever trying to question them.

Oh I didn't just "buy" them. As a low-level programmer who's suffered for a long time from intrinsic inefficiencies and C++, I became a compiler and runtime engineer working on the JVM to solve the problems I had in C++.

> This is the "wasted RAM is unused RAM" mentality and it doesn't work, because you usually have multiple competing programs and when you run out of RAM, your system will start swapping

No, it's actually more involved and interesting than that, but you'll have to wait for my talk.

> Ok, now you're just writing nonsense. Nowadays people have CPUs with multiple cores and use an OS with a scheduler. If you have two programs taking up 100% of the CPU, the OS will give each process some of the hardware resources. You can't just assume some 100% CPU blockage here just because it is convenient for your argument

I didn't. I specifically said it was just an example to demonstrate the inter-relatedness of RAM and CPU since accessing RAM requires CPU. To understand why every single language that can isn't limited by other constraints and has the engineering resources to do so uses the same basic memory management algorithm as Java I guess you'll have to watch my talk when it's published.

> Again, this is completely incorrect in so many ways that you're bragging you know nothing about how modern computers work.

Wow. I guess it doesn't take much to be an engineer working on safety critical realtime applications and then on one of the worlds most advanced optimising compilers and you can get pretty far without knowing how computers work.

> CPU cores have their own local memory resources called caches. Depending on how your code is written, you may tile your data so it fits entirely in cache and operate within the local memory.

The data you need to access at any one time and the overall memory consumption of your program are two very different things. Maybe you don't know this, but CPU caches don't work by caching a large contiguous portion of the address space.

> When performing inter thread communication, there are often situations where the data often doesn't even get written and then loaded to main memory, since atomic operations can make use of the MESI cache coherency protocol to pull the data directly from another cores' cache.

I find it hilarious that you're trying to teach me about MESI, given that designing algorithms and data structures that are efficient on top of MESI was one of my jobs [1], and I advised Intel on architecture, but okay, maybe I know nothing about computers, as you concluded from a paragraph where I tried to give people who may not be compiler or memory management experts some intution about modern memory management design.

FYI, modern malloc/free allocators are also intentionally less footprint-optimised than older ones to get better performance (although they can't offer all the optimisations of moving collectors because they're not allowed to move pointers), but maybe none of the people writing the compilers or memory management mechanisms you use know computers as much as you do, and you know all there is to know.

[1]: I later even wrote, for a general audience, about data structures over distributed MESI (well, MOESI to be precise) protocols: https://highscalability.com/the-performance-of-distributed-d...

I am not GP poster. I find pron points interesting even if I work in the gamedev on game engines. If you don't mind I will try to explain how I see them interesting. Since I have not worked on Rust systems I will stick to C++.

Note his example elsewhere in this discussion of 2 projects done at same time in Java and Rust and the complaint that Rust system used too many locks. This can happen in C++ too. But why it does not happen in (my) practice? Because C++ evolved to not use locks in large scale parallel systems. This was said from mainstage conferences keynotes at least since 2013 [1]. So there is "normal C++" and "C++ that works at large scale" and they are not the same C++ languages. The performance scales between them are many orders of magnitude. Imho it does not mean that Java anywhere near the best of what C++ can do. So here we are talking past each other. pron is correct that Java is not bad and you are correct that you have no reasons to leave Rust.

1. https://sean-parent.stlab.cc/presentations/2013-09-11-cpp-se...

Wow looks great!!!