Global Memory Shortage Crisis: Market Analysis • Hacker News Board

Re: Global Memory Shortage Crisis: Market Analysis

#101

Post by Mr_Minderbinder » Mon, Dec 29, 2025, 9:32 AM UTC

0xbadcafebee wrote: post #96

PostOnce wrote: post #82

Earlier quoted context omitted.

Developers would prefer to write good software, the challenge and the craftsmanship are a draw. However, the customers do not care and will not pay more so the business cannot justify it most of the time. Who will pay twice (or five times) as much for software written in C instead of Python? Not many.

Well this is patently false. For the past 3 decades, programmers have intentionally made choices which perform as poorly as the hardware will allow them. You can pretty much draw a parallel line with hardware advancement and the bloating of software. It hasn't gotten 100x harder to display hypermedia than it was 20 years ago. Yet applications use 10x-100x more memory and CPU than they used to. That's not good softwar…

> You can pretty much draw a parallel line with hardware advancement and the bloating of software.

I do not think it is surprising that there is a Jevons paradox-like phenomena with computer memory and like other instances of it, it does not necessarily follow that this must be a result of a corresponding decline in resource usage efficiency.

Re: Global Memory Shortage Crisis: Market Analysis

#102

Post by imtringued » Mon, Dec 29, 2025, 1:52 PM UTC

zozbot234 wrote: post #48

Youden wrote: post #45

"However, this is not just a cyclical shortage driven by a mismatch in supply and demand, but a potentially permanent, strategic reallocation of the world’s silicon wafer capacity. [...] This is a zero-sum game: every wafer allocated to an HBM stack for an Nvidia GPU is a wafer denied to the LPDDR5X module of a mid-range smartphone or the SSD of a consumer laptop." I wonder if this will result in writing more memory-…

There's plenty of scope for local AI models to become more efficient, too. MoE doesn't need too much RAM: only the parameters for experts that are active at any given time truly need to be in memory, the rest can be in read-only storage and be fetched on demand. If you're doing CPU inference this can even be managed automatically by mmap, whereas loading params into VRAM must currently be managed as part of running a…

MoE works exactly the opposite way you described. MoE means that each inference pass reads a subset of the parameters, which means that you can run a bigger model with the same memory bandwidth and achieve the same number of tokens per second. This means you're using more memory in the end.

Re: Global Memory Shortage Crisis: Market Analysis

#103

Post by juancn » Mon, Dec 29, 2025, 2:14 PM UTC

I wonder if hurting the consumer space won't end up crippling the AI hyperscalers ambitions?

I mean, the lack of affordable consumer hardware may end up further reducing the need for AI.

On the other hand, it may end up shifting workloads to the cloud instead.

Heck, time will tell.

Re: Global Memory Shortage Crisis: Market Analysis

#104

Post by electroly » Mon, Dec 29, 2025, 3:22 PM UTC

hinkley wrote: post #34

One of the things I’ve been hoping for every time a new EC2 instance comes out is for them to unpin the memory:core ratio a bit. I don’t expect they have enough r# and c# users to completely balance things out so what they’re really doing is selling people more CPUs to get the memory they need. It would be nice if it were creeping up generation to generation. But if this keeps up I fear the opposite.

Note: the scale does go further than "r" on the high-memory end with some specialty "x" families.

    c*: 2GB per vCPU
    m*: 4GB per vCPU
    r*: 8GB per vCPU
    x2idn/x8g: 16GB per vCPU (!)
    x2iedn/x2iezn/x8aedz: 32GB per vCPU (!)

Re: Global Memory Shortage Crisis: Market Analysis

#105

Post by hinkley » Mon, Dec 29, 2025, 9:57 PM UTC

electroly wrote: post #104

hinkley wrote: post #34

One of the things I’ve been hoping for every time a new EC2 instance comes out is for them to unpin the memory:core ratio a bit. I don’t expect they have enough r# and c# users to completely balance things out so what they’re really doing is selling people more CPUs to get the memory they need. It would be nice if it were creeping up generation to generation. But if this keeps up I fear the opposite.

Note: the scale does go further than "r" on the high-memory end with some specialty "x" families. c*: 2GB per vCPU m*: 4GB per vCPU r*: 8GB per vCPU x2idn/x8g: 16GB per vCPU (!) x2iedn/x2iezn/x8aedz: 32GB per vCPU (!)

Yeah those are pretty spendy. I know one comes with extra guaranteed bandwidth which is kind of handy if you’re sharing a small number of cache nodes among a lot of servers. But we were doing okay running r6 for cache, though my coworker who knew the ritual for migrating them did eventually get a little boost out of switching us to r7’s. The latency wasn’t great and I don’t think faster network cards would have helped that. There was already plenty of incentive for us to do per-request promise caching to avoid pulling the same keys multiple times in a request but that was necessary because the business model forced the architecture to tolerate nondeterminism. The cost per request was what eventually killed them (the economy dipped and customers ran to cheaper vendors), but I’ve never seen a company survive being stupid for as long as this place did.

Well, except IBM. Maybe Yahoo.

Re: Global Memory Shortage Crisis: Market Analysis

#106

Post by ragequittah » Mon, Dec 29, 2025, 11:28 PM UTC

ganoushoreilly wrote: post #80

0xbadcafebee wrote: post #67

Earlier quoted context omitted.

> I wonder if this will result in writing more memory-efficient software? If the consumer market can't get cheap RAM anymore, the natural result is a pivot back to server-heavy technology (where all the RAM is anyway) with things like server-side rendering and thin clients. Developers are far too lazy to suddenly become efficient programmers and there's plenty of network bandwidth.

This is by design. Rent your computer.. don't buy! Use Geforce Now!

There is a small part of me that wonders if my $3000 computer is worth it when that could get me about 12 years of geforce now gaming with an updated graphic card and processor at all times. But I like to tinker so I'll probably end up spending $10k or more by the end of that 12 years instead.

Re: Global Memory Shortage Crisis: Market Analysis

#107

Post by AtlasBarfed » Thu, Jan 01, 2026, 8:26 PM UTC

Youden wrote: post #45

"However, this is not just a cyclical shortage driven by a mismatch in supply and demand, but a potentially permanent, strategic reallocation of the world’s silicon wafer capacity. [...] This is a zero-sum game: every wafer allocated to an HBM stack for an Nvidia GPU is a wafer denied to the LPDDR5X module of a mid-range smartphone or the SSD of a consumer laptop." I wonder if this will result in writing more memory-…

I'm waiting for the good AI powers software.... Any day now.

Ideally, llm should be able to provide the capability to translate from memory inefficient languages to memory efficient languages, and maybe even optimize underlying algorithms in memory use for this.

But I'm not going to hold my breath