The RAM shortage could last years

> In the case of Rust, it has Arenas, stack allocated structs, and generics via monomorphization. Not only can you avoid both of these things, it doesn't even seem that difficult. If you're saying the borrowchecker just becomes too cumbersome to do for sufficiently large applications, that's fine.

Not really.

First, let's look at stack-allocated structs and think about how much data can live in them. The typical stack size is 2MB, but because the only live data in stacks are in caller functions, we can say that on average, the amount of live data that a stack holds is 1MB. Now, look at how much RAM an application uses in MBs and divide it by the number of threads. Usually, the ratio is much higher than RAM, which means that data in stacks is not a significant portion of the program's data. (Async changes this calculus a bit, but async is extremely limited in Rust as it doesn't allow recursion, FFI, or dynamic dispatch; proper user-mode threads, like the ones in Java and Go actually make stack allocation more useful.)

Now let's look at arenas. Arenas are extremely efficient because they offer a similar RAM/CPU tradeoff knob as moving GCs, but they're not as general (if your allocation pattern supports them well, they're great, but you can't generally use them). But in Rust, things are much worse, because arenas are quite limited; too many standard-library data structures, including strings, vectors, and maps can't easily be plugged into an arena. The only language that gives you arenas' full power (which, again, is not completely general) is Zig. This is one of the reasons hardcore low level programmers find Rust so underwhelming (the other being that too many things that are important in low-level programming, including basic data structures but also benign concurrency, require unsafe).

> But they got to the same performance as Java with less RAM, at the cost of dev experience.

You say "dev experience" as if it's some quality-of-life thing. They traded off a cheap resource, RAM, for an eternal maintenance and evolution cost that would only grow higher as the program grows. And remember that RAM isn't entirely fungible. It's hard to get less than 1GB per core (either in bare metal or in cloud VMs/containers) so using less RAM often saves you $0.

> but for a smaller app, that could actually be a worthy tradeoff, no? Like...a desktop app?

I said that low-level languages can offer good performance in small programs, but many desktop apps aren't small. Claude Code's CLI is over 500KLOC.

> So all we have is anecdotal experience that Java is more performant for large, long lived processes with many contributors

If anything, benchmarks are much more anecdotal. Not only are there fewer benchmarks than applications, but they don't even resemble real programs. But yeah, ever since operations lost their intrinsic costs some 20 years ago - with CPU cache hierarchies, branch prediction, and ILP, more powerful optimising compilers, and more elaborate GCs/memory allocators, the ability to generalise from one program to another is close to nil. So yeah, experience is all we have to go with. Going with the numbers we have (some benchmarks that don't extrapolate) rather than the numbers we need but don't have doesn't help.

I can tell you that the loss of intrinsic operation costs has made our lives as compiler/runtime developers much harder, because we can no longer tell people that this operation is generally fast or generally slow. But that doesn't change the fact that this is our reality.

> what the original claim was, which is the claim you cannot reduce RAM consumption without harming CPU utilization

No. I wrote, and I quote: "Using a lot less RAM often implies using more CPU."

> That's a really strong claim considering we have painted a much narrower scope of when that's true.

Did we? Most of software (measured by the distribution of paid programmers) is in large applications.

> Most of us aren't working on 3M LOC faang apps. We are working on smaller things, or desktop apps

I don't think that's true at all. Forget FAANG. Most software isn't written by software companies at all, but is in-house software (well, Netflix isn't a software company, so I guess it's one FAANG letter). The bulk of software is in things like telecom management and billing, banking and finance, manufacturing control, logistics and shipping, healthcare and hospitality, retail and payment processing, travel, government, defence. 3MLOC is quite typical. People who work on smaller software are overrepresented in Silicon Valley and, I'm guessing, among HN readers, but they're the outliers.

> Grandma doesn't have the vocabulary to complain about RAM, true. But her computer is slow, she asked her grandson for help, and her grandson told her to use Spotify in the browser, not download the app. And now she has to be mindful of what she has open, even though 8gb of RAM is actually a lot, we've just lost sight of it.

Does she, though? I doubt she's running anything intensive in the background, so she's really only using one program at a time, and SSDs are fast enough these days to page in virtual memory when she switches programs, unless the one program she's currently using eats up the 8GB. I agree that if her OS - the one thing she needs to run in the background - is taking up a lot of RAM that could be a problem, but the OS is special. Her computer is slow not because shes using a program that eats up a lot of RAM, but because she's inadvertently running a lot of stuff in the background that shouldn't be running at all (browser plugins? some programs that add themselves as login items?) A Surface Laptop comes with 16GB of RAM. No single program uses even half of that.

> The problem is CPU and RAM usage are fundamentally different. I don't get to know what will be run with my program, so I don't get to know how restricted RAM is.

You'd think that, but that's not the case. I admit that I only recently started thinking deeply about this, thanks to some conversations with a colleague who's one of the world's leading experts on memory management, and it was so eye-opening that I gave a talk about this at the recent Java One (because my colleague wasn't available). There are two sides to this:

1. On the demand side, the key is that the use of RAM necessitates the use of CPU (and vice versa): writing and reading to/from RAM requires CPU, but also we write to RAM only when we expect the program to read it in the future. This means that any CPU we use, takes away the ability of another program to use some RAM (because using RAM requires CPU). To give the basic intuition for this, I mentioned the extrme example of a program that uses 100% of CPU. Such a program effectively captures 100% of RAM no matter how much of it it actually uses because no other program can use any RAM, as no other program has the CPU available to access it. You don't need to know anything about what other programs do. Another way to think about this is that the machine is spent whenever the first of RAM and CPU is exhausted.

2. On the supply side, RAM and CPU - whether in metal or in virtualised hardware - are effectively sold as a package (it's hard to get less than 1GB per core, except on embedded devices these days). Furthermore, both moving GCs and (to a far lesser extent) memory allocators can trade RAM and CPU (sophisticated memory allocators aren't quick to return RAM to the OS and maintain internal buffers).

So even though it is true that different programs may have different CPU/RAM usage patterns, you have to think about the ratio rather than CPU and RAM in isolation, and try to achieve some approximate balance. To put it simply, if a program uses a lot of CPU it doesn't make sense for it to use little RAM, because by using a lot of CPU it is effectively depriving other programs of their ability to use RAM (as that requires CPU). There are some exceptions, such as large caches, but the tradeoffs there are very different and too complicated to go into here (I did cover that in my talk).

> It's purely a dev experience decision because people don't know how to ship desktop apps. It's pure waste for the user.

No. I mean, some of it is probably waste, but:

1. What you call "dev experience" also affects the user because it directly impacts the cost of software. Users want cheaper software.

2. More relevant to this particular discussion is what else the user could do. Having "more programs open" isn't a problem thanks to SSDs and virtual memory. So we're talking about programs that are actively using the CPU for something, and they, too, need a balance of the RAM/CPU ratio.

I'm not trying to be dogmatic in the other direction and assert that Electron is necessarily the best tradeoff. But I'm saying that efficiency is ultimately about money that is spent on a combination of RAM, CPU, and software, and when you look at the full picture you see that it's more complicated than it seems. It's not that the software industry has decided to waste users' money. If it did, there would be a competitive edge to programs that use less RAM, but we don't see that competitive edge. What we do see is a few people on HN saying how they simply can't live with VS Code's 50ms keystroke latency and how amazing is some other editor with only 20ms latency that's likely to go out of business soon [1]. The people who made these decisions aren't some early-career developers who just like hot code reloading or some such.

[1] Yes, I do think Rust is more hardware-efficient than JS, but here I'm looking at an even bigger picture. And yes, if you rewrite from JS to C++/Java/Rust/Go you can win on hardware, but as I said at the very beginning, any such rewrite is not really "an optimisation".

> To put it simply, if a program uses a lot of CPU it doesn't make sense for it to use little RAM

The fallacy in that reasoning is that a program that's using a lot of CPU (especially if it's a huge MLOC-sized app) is most likely using up its CPU on memory throughput, not pure number-crunching compute! So at least for the enterprise app case (not pure number crunching), you'd actually need a tunable tradeoff between memory throughput and total RAM footprint, and adopting copying/moving GC's just doesn't give you that. Collections cycles are a huge burden on memory throughput: thus, indirectly, on the very thing you're calling "CPU". The theoretical prospect of winning by forgoing collections cycles outright (pure bump arena allocation) is explicitly excluded here since we're talking about long-running programs that will at some point need to garbage collect.

Heap allocation may have marginally higher "CPU" use in the pure compute sense, but that's exactly the kind of CPU use that does trade off successfully with a lower RAM footprint.

Similarly, non-moving concurrent garbage collectors like Go's also successfully navigate this tradeoff compared to moving/copying collectors, because their collection work, while compute- and to some extent memory-traffic intensive (though less so than if copying/moving memory was involved!) can be largely (though not completely - some minor compute overhead on the hot path is still present) shunted off to a lower-priority background thread.

On the other side of the tradeoff, arenas and caches increase memory footprint in a way that's low-impact on memory throughput (unlike pervasive use of a copying/moving GC) because only live data is accessed as needed, and deallocating the arena is a single operation. The tradeoff is actually highly favorable to low-level languages, which commonly use arenas to manage challenges with heap allocation such as fragmentation.

I feel like this has been a great discussion but all the good technical talk is tapering off. There's more rhetorical semantics now than anything. But I appreciate you teaching me. I'm a bit tired in this reply. I wish I could find you a few reports from people on here basically renouncing Java because they could not optimize it any further after 20 years programming in it, and moving to Rust. I'd be curious what you'd think.

> You say "dev experience" as if it's some quality-of-life thing. They traded off a cheap resource, RAM, for an eternal maintenance and evolution cost that would only grow higher as the program grows No, I say dev experience because I brought it up earlier, and staked my claim on it. Because it's an umbrella term that covers nice-to-haves and how ergonomic the language and ecosystem are. The antithesis would be lots of repetitive plumbing that slows down feature release. It's one part of the triangle. They now are shipping way behind but wound up with a product that is just as fast and uses less RAM.

... and will cost much more to evolve, costs will never drop and may well rise. These costs are higher than the RAM they saved, which was free anyway, because it couldn't be used for anything else.

> That kind of decision can matter to other projects. Smaller projects likely wouldn't have such a slow turnaround. Again, we're not advocating for everyone to do rewrites. Threads like this are people begging app developers to stop using stuff not appropriate for desktop apps.

The people begging are sometimes right and sometimes really wrong. They are sensitive to certain things but don't consider the full picture.

> I said that low-level languages can offer good performance in small programs, but many desktop apps aren't small. Claude Code's CLI is over

> Couldn't tell you why that tool needs to use that much, but most desktop apps don't. They are built on vendor code and keep the actual app code small, and that makes them excellent candidates for what we're talking about.

VS Code is something like 5MLOC. Slack is probably around a million. Every desktop app (that isn't bundled with the OS) that I or people I know use are roughly that size or bigger.

> What you're describing is an extremely narrow class of software that most people will never touch. But I think it sounds cool.

First of all, this is the class of software that most people rely on the most by far. It certainly contributes more economic value than other software. You use that software every time you tap your bank card; every time you place or receive a call or send or receive a text message; every time a package arrives at your doorstep; every time you watch any video on any platform; every time you receive medical treatment or stay at a hotel. Clearly, it's the class of software that contributes the majority of value that software delivers. I don't know exactly how many people work on such software. I think around 50% of developers at least, but even if I'm wrong, we're talking at least a few million developers.

> but at least they measure something, and it's public and reproducible.

There is zero value to a measurement that is not relevant to you, and negative value to making you think it's relevant ("well, it's not the number I need but it's some number I have so I'll go by that") when it's not.

> Anecdotes are vague stories that are impossible to evaluate.

You can evaluate them at least as well as you can benchmarks - by asking questions that will allow you to know if the information is relevant to your use case - only they at least have a chance of being relevant, while benchmarks really rarely are. I will just say that if you don't know how exactly a memory allocator is implemented (e.g. whether and how it degrades with time), it is absolutely impossible for you to evaluate a benchmark that purports to measure memory management. There is nothing you can learn from it because you don't really know what it is that's been measured.

> I had an anecdote of someone saying they can't use Java anymore because, even with 20 years of experience, they cannot optimize Java any further for what they need. They rewrote it in Rust. It works much better now, and it's not even close. What am I to make of that?

You are to make of it that, in the past 20 years at least, we have no ability to extrapolate from one program to another. Given that languages like C++, Rust, Java, Zig, and C# are all at the topmost performance category, it makes a lot of sense that in some situations X will be faster than Y and in others Y will be faster than X.

> There are absolutely apps that run 8gb of RAM. And pageswaps are not good, even with an SSD. They're a real problem.

I've not seen that in quite a few years now. Right now I have Chrome open on four streaming platforms. Just over 500MB. I have over 100 tabs open in Safari. About 1.2GB. I'm sure there are some apps that use 8GB of RAM, but these aren't apps that grandma uses or wants to use.

> That's a cool realization. But I think it's slippery. Only in extreme scenarios will your CPU actually block RAM. The 100% usage scenario makes sense.

No. This scales. Again, every CPU cycle you consume takes a cycle away from some other program and reduces its ability to use RAM (as that requires the cycle).

> But most of the time, your CPU is going to be underutilized and capable of letting every app use RAM freely.

No. Using RAM means using CPU. I you're idle, then you're not using RAM. Might as well be paged to SSD. You get zero points for keeping RAM significantly more plentiful than free RAM. You save $0.

> How? If you're forced to use work software, you have no competition to go to. Same for your music app, your social network, your team's chat tool. The "competitive edge" argument requires users to actually have a choice, and for most desktop software they don't they use what their employer, school, or social network has standardized on. Where users do have free choice, they gravitate toward leaner options constantly. Sublime kept paying customers against free Electron alternatives. Mobile platforms enforce resource discipline and have no Electron equivalent. The competitive edge for leanness exists; it just can't express itself when ecosystem effects lock users in.

The competitive edge does not require users to have a choice. It requires a real edge. If some software really makes you more productive, then your employer would be foolish not to buy it. If some software allows the school to significantly save on hardware - the same. The reason these products don't catch on is because they're written by hackers with certain sensitivities who do not understand the economics of their users' hardware and software.

> So I still feel strongly there are cases where you could make sufficiently small programs in Rust that wouldn't devolve into spaghetti. That would give you great performance and ram usage.

Sure. Like I said, low-level programs are fast and efficient for small programs. But even then, you need to look at the full picture to know how much, if any, money you're saving.

> The fallacy in that reasoning is that a program that's using a lot of CPU (especially if it's a huge MLOC-sized app) is most likely using up its CPU on memory throughput, not pure number-crunching compute!

No, there's no such fallacy here because that assumption is not needed for the conclusion. The point is that CPU is needed to use RAM, and so if you use CPU for whatever reason - even to loop for an hour over some integers - you are consuming a resource that is needed to use RAM (by another program). So the use of CPU "captures" RAM whether it uses RAM or not, so it might as well use it.

The extreme example I gave was that a program that uses 100% CPU (again, even if it uses zero RAM) effectively capures 100% of RAM because no other program can use any RAM while that program is running. This extreme example is just to build some intuition, but it scales to lower CPU utilisations.

> Collections cycles are a huge burden on memory throughput

I don't even know where to start. The whole point of moving collectors is that the can make the cost of memory management arbitrarily low, reduing the overhead compared to free list approaches. A collection cycle does a constant amount of work (per program & workload), but the frequency of the collections can be made arbitrarily low. This is memory management 101.

The problem of moving collectors has traditionally been the impact on latency, not on throughput (they were always better on throughput than free lists) - until the advent of pauseless moving collectors.

> Heap allocation may have marginally higher "CPU" use in the pure compute sense, but that's exactly the kind of CPU use that does trade off successfully with a lower RAM footprint.

Except it doesn't given the actual economics of RAM and CPU. You'll need to wait for my talk to be posted to YouTube (I can't reproduce it all here), but in the meantime you can watch this one, by my colleague, which was a keynote at the most recent ISMM (International Symposium on Memory Management): https://youtu.be/mLNFVNXbw7I

The problem is that Erik is one of the world's leading experts on memory management, and he's talking to other experts, so his talk assumes quite a bit of familiarity with the subject. Also, his comparison focuses on tracing GCs, leaving the one to malloc/free implicit.

> Similarly, non-moving concurrent garbage collectors like Go's also successfully navigate this tradeoff compared to moving/copying collectors

Again, except they do not. Go users experience severe problems with the GC that Java users no longer do precisely because of the inefficiencies of their simple GC (the JDK used to have such a GC, but we removed it five years ago when newer, more sophisticated algorithms yielded better results.

> On the other side of the tradeoff, arenas and caches increase memory footprint in a way that's low-impact on memory throughput (unlike pervasive use of a copying/moving GC)

This is simply not true, and shows unfamiliarity with how modern moving collectors actually work (remember that the first open-source high throughput, paseuless moving collector was first released two and a half years ago). Moving collectors offer pretty much the same tradeoff as arenas. The key points are:

1. A generational design makes copying a relatively rare operation to begin with (only a relatively small number of objects are copied).

2. The frequency of collections can be made arbitrarily low.

If the usage happens to be arena-like, i.e. no objects survive, nothing is copied (unrelated long-lived objects are already in the old gen, and because the old gen is untouched, there's no need to compact anything there).

The reasons for not using moving collectors have nothing to do with throughput:

1. Latency used to suffer. Low-latency moving collectors were a very advanced technology. The first open-source one is younger than ChatGPT.

2. Moving collectors impact the design of FFI (with C, etc.), as C (etc.) does not support moving pointers for reasons having nothing to do with performance. It's very hard for languages that want a very direct and simple FFI (as FFI is very common in code) have a hard time implementing efficient moving collectors.

3. Good moving collectors require large expert teams (let alone pauseless moving collectors). The languages that have them (to varying degrees of sophistication and performance) are well funded ones. In particular, they are the JDK team, the .NET team, and the V8 team. Good allocators (for malloc/free) are also big and sophisticated beasts these days, but they're much easier to reuse in different languages (i.e. not only by C/C++/Zig/Rust, but also by Python). Effectively, large pieces of those languages' runtime is "offshored" to unrelated specialist teams.

> The tradeoff is actually highly favorable to low-level languages, which commonly use arenas to manage challenges with heap allocation such as fragmentation.

This is also not true (and I say this because I'm primarily a low-level programmer, and have been doing low-level programming for over 25 years). First, C++ and Rust in particular make arenas hard to use to their full power (which is one of the several reasons low-level programmers prefer Zig). Second, if you're not familiar with the severe costs of memory management in low level languages, I can only conclude you haven't been doing it for very long.

Java was designed, among other things, to reduce the severe and hard-to-fix performance problems that many C++ programs had experienced (and do to this day). The hard problems concern both the limitations of AOT compilation and of free-list-based memory management. I'm not saying all C++ programs suffer from these issues, but a huge class of them do, which is one of the reasons large programs have migrated to Java.

The problem with benchmarks isn't that they themselves are lying. Benchmarks always tell the truth - about themselves. The problem is in the conclusions people draw from them. In the nineties benchmarks were still a little extrapolatable because we could say X is slow and Y is fast, as many operations had an intrinsic cost. These days, almost no benchmark (certainly microbenchmark) is extrapolatable to anything beside itself. Is a branch slow or fast? That depends on what the program did before and what it intends to do later. Is memory access slow or fast? Ditto. Function call? Allocation? They're all so context-dependent now that the only use of benchmarks of some mechanism is for the authors of the mechanism who know exactly how it works, what exactly is being measured, and what can be extrapolated from that.

If I write a malloc benchmark I may think, oh, this measures the cost of malloc/free. In reality, it only measures the cost for a program whose concurrency, allocation/deallocation patterns, and duration match exactly what I wrote, and bear little resemblance to the numbers I'd get if any of those were different.

So I'm not saying that the Benchmark Game is lying. It is telling the truth about how long those programs ran. It's just that what we can generalise from those benchmarks is even less than what we can from more "interesting" ones, but given that even that is close to nothing anyway, maybe it doesn't matter.