Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The RAM shortage could last years

theverge.com

Post reply on HN
Print view View original

441–450 of 536 posts

Re: The RAM shortage could last years

#441

Post by dannersy »

romanovcode wrote: post #422
dannersy wrote: post #373

Earlier quoted context omitted.

> The usage of LLMs is continuing to increase ~exponentially I would like a source for that statement. Additionally, I want to know by who? Because it certainly isn't end users. Inflating token usage doesn't make it any more economically viable if your user base, b2b or not, hasn't increased with it. On the contrary, that is a worse scenario for providers.

> Additionally, I want to know by who? 1. As a consultant pretty much every company I have worked with in the last 2 years are doing some kind of in-house "AI Revolution", I'm talking making "AI Taskforce" teams, having weekly internal "AI meetings" and pushing AI everywhere and to everyone. Small companies, SMEs and huge companies. From my observation it is mainly due to C-level being obsessed by the idea that AI wi…

Yeah, my only hope is that this is unsustainable, admittedly for selfish reasons.

I know plenty of engineers being forced to use these tools whether they want to or not. A lot of which are okay with using AI liberally, but don't particularly like generative AI and see it as pretty irresponsible (which feels more true by the week and it is clear from first hand experience). I don't know, there is a huge gradient of users, but I would argue that in previous revolutionary technologies, we didn't have to force people to use a good tool. I didn't have to be forced to use Google search or Google Maps, tech that is now ubiquitous with western society. It seems really suspect that suits have to enforce the use of something that is supposed to change the way we work and be a force multiplier.

Re: The RAM shortage could last years

#442

Post by mitjam »

torginus wrote: post #42
stuxnet79 wrote: post #11

Ok so Samsung, SK Hynix and Micron do not have the capacity to meet demand. Also, what little capacity they do have they are allocating to HBM over DRAM. Based on my limited knowledge HBM can not be easily repurposed for consumer electronics. Translation: main street is cooked for the next 3-4 years. It doesn't stop there though. OpenAI is currently mired in a capital crunch. Their last round just about sucked all th…

The Radeon VII came out in 2019 as a $700 consumer GPU with an 1TB/s HBM2 memory subsystem which is more than any consumer GPU you can get today, including the high-end ones afaik. At that point in time, there was a whole lineup of AMD GPUs with HBM going down into the midrange. If they could make this stuff and sell it to regular people a decade ago for very palatable prices, why do they come up with the idea that t…

Yes, it's interesting that HBM was invented by a collaboration between AMD and SK Hynix. It seems, HBM is the way to go for GPUs, anyway.

The GB202 die that's in the GDDR7 based RTX 5090 and RTX 6000 Pro literally needed to be this big to support the 512bit memory bus. It's probably only getting worse with smaller node sizes. (see https://www.youtube.com/watch?v=rCwgAGG2sZQ&t=65s).

BTW: The 1TB/s is matched by RTX4090 and surpassed by the RTX5090 (1,79 TB/s).

Re: The RAM shortage could last years

#443

Post by pron »

zozbot234 wrote: post #429
pron wrote: post #424

Earlier quoted context omitted.

> GC burns far more CPU cycles Far less for moving collectors. That's why they're used: to reduce the overhead of malloc/free based memory management. The whole point of moving collectors is that they can make the CPU cost of memory management arbitrarily low , even lower than stack allocation. In practice it's more complicated, but the principle stands. The reason some programs "avoid the heap like the plague" is be…

Moving collectors as generally used are a huge waste of memory throughput, and this shows up consistently in the performance measurements. Moving data is very expensive! The whole point of ownership tracking in programming languages is so that large chunks of "owned" data can just stay put until freed, and only the owning handle (which is tiny) needs to move around. Most GC programming languages do a terrible job of…

That's just not true. To give you a few pieces of the picture, moving collectors move little memory and do so rarely (relative to the allocation rate):

In the young generation, few objects survive and so few are moved (the very few that survive longer are moved into the old gen); in the old generation, most objects survive, but the allocation rate is so low that moving them is rare (although the memory management technique in the old gen doesn't matter as much precisely because the allocation rate is so low, so whether you want a moving algorithm or not in the old gen is less about speed and more about other concerns).

On top of that, the general principle of moving collectors (and why in theory they're cheaper than stack allocation) is that the cost of the overall work of moving memory is roughly constant for a specific workload, but its frequency can be made as low as you want by using more RAM.

The reason moving collectors are used in the first place is to reduce the high overhead of malloc/free allocators.

Anyway, the general point I was making above is that a machine is exhausted not when both CPU and RAM are exhausted, but when one of them is. Efficient hardware utilisation is when the program strikes some good balance between them. There's not much point to reducing RAM footprint when CPU utilisation is high or reducing CPU consumption when RAM consumption is high. Using much of one and little of the other is wasteful when you can reduce the higher one by increasing the other. Moving collectors give you a convenient knob to do that: if a program consumes a lot of CPU and little RAM, you can increase the heap and turn some RAM into CPU and vice versa.

Re: The RAM shortage could last years

#444

Post by nilkn »

irishloop wrote: post #268
Mawr wrote: post #253

Earlier quoted context omitted.

> and at that time the prevailing wisdom was to never buy a pre-built That's still the case, and always will be — with a pre-built you're at the very least paying for someone to assemble it for you, so it's always going to be more expensive as a baseline. Beyond that, the chance they've chosen good components and haven't tried to screw you over on less flashy ones like the motherboard and power supply is low. That's…

> with a pre-built you're at the very least paying for someone to assemble it for you, so it's always going to be more expensive as a baseline Except isn't it possible that pre-built companies actually get better deals on hardware bought in bulk, and therefore could offset the labor costs with cheaper materials?

I believe this is exactly what's going on -- they're buying parts in bulk, often months in advance, and locking in deals that a single consumer can't easily go get on the open market right now.

Hardware pricing and availabilty pre-COVID was pretty predictable and stable, which meant the consumer could extract a meaningful cost advantage if they were willing to do the relatively modest amount of work of sourcing components individually and personally assembling the build. Right now, though, some places like Microcenter appear to have a cost advantage that fundamentally relies on market and pricing instability and can only be achieved through deeper integration with the supply chain and bulk purchasing in advance -- something a retailer like Microcenter can do, but I personally cannot.

Re: The RAM shortage could last years

#445

Post by [deleted] »

Mawr wrote: post #253
nilkn wrote: post #224

As an aside, recently I wanted to refresh my gaming PC, but the price shock and general lack of availability of buying components individually made it seem hardly worth it, so I just kept deferring the project. Then, mostly by chance, I saw that my local Microcenter had some pre-builts for sale, and I ended up picking one up for The last time I built a gaming PC was upwards of a decade ago, and at that time the preva…

> and at that time the prevailing wisdom was to never buy a pre-built That's still the case, and always will be — with a pre-built you're at the very least paying for someone to assemble it for you, so it's always going to be more expensive as a baseline. Beyond that, the chance they've chosen good components and haven't tried to screw you over on less flashy ones like the motherboard and power supply is low. That's…

[deleted]

Re: The RAM shortage could last years

#446

Post by matt-p »

AlecSchueler wrote: post #436
AnthonyMouse wrote: post #118

Earlier quoted context omitted.

Whether it's generally a reasonable place to build them isn't the percentage. The number seems to be ~3%.

3% of the global total does seem surprisingly significant.

London is about 3X bigger than Amsterdam in terms of capacity. If you look at the core western europe market (so called FLAP-D) which I am rather familiar with London holds 35% of that market and Amsterdam about 12%. I'm not at all sold that the old rule of thumb that this market is about 25% of global capacity has been true since about 2023 because of the AI buildout in NA/ME.

Re: The RAM shortage could last years

#447

Post by ethersteeds »

nilkn wrote: post #224

As an aside, recently I wanted to refresh my gaming PC, but the price shock and general lack of availability of buying components individually made it seem hardly worth it, so I just kept deferring the project. Then, mostly by chance, I saw that my local Microcenter had some pre-builts for sale, and I ended up picking one up for The last time I built a gaming PC was upwards of a decade ago, and at that time the preva…

> I ended up picking one up for I'm struggling to put this in context. For comparison, what was your budget for refreshing the pc you had? Were the planned upgrades going to exceed $5k at current prices? Or is the situation that a pre-build machine with far better components was now only marginally more?

Or is it that pre-built gaming PCs have stopped being a joke? I had the experience building a bicycle: I was certain I was taking the frugal path sourcing each component individually and putting it together myself. At the end I was horrified to realize I spent far more than a new bike with superior components. It was pointed out that bicycle makers are buying by the pallet and will beat diy every time — so long as they're building something I want to buy.

Re: The RAM shortage could last years

#448

Post by romanovcode »

dannersy wrote: post #441
romanovcode wrote: post #422

Earlier quoted context omitted.

> Additionally, I want to know by who? 1. As a consultant pretty much every company I have worked with in the last 2 years are doing some kind of in-house "AI Revolution", I'm talking making "AI Taskforce" teams, having weekly internal "AI meetings" and pushing AI everywhere and to everyone. Small companies, SMEs and huge companies. From my observation it is mainly due to C-level being obsessed by the idea that AI wi…

Yeah, my only hope is that this is unsustainable, admittedly for selfish reasons. I know plenty of engineers being forced to use these tools whether they want to or not. A lot of which are okay with using AI liberally, but don't particularly like generative AI and see it as pretty irresponsible (which feels more true by the week and it is clear from first hand experience). I don't know, there is a huge gradient of us…

From my limited experience in multiple companies, as stated before I see one very common pattern - The process from feature idea to development is just bad. PMs do not know what exactly they want. C-level interjects in the middle and changes requirements. QAs are unsure what to test because acceptance criteria is vague.

C-level strongly believes that AI will fix all these issues. They believe that AI will fix their broken processes.

I see strong resemblance with "Agile Development" ~15 years ago. Extremely hyped, noone asked if their org even is a fit for it or need it, and most importantly - the only way to fix agile is to do more agile. Same with AI right now.

Re: The RAM shortage could last years

#449

Post by FuckButtons »

rekabis wrote: post #121
cbdevidal wrote: post #96

I’m a bit of an optimist. I think this will smack the hands of developers who don’t manage RAM well and future apps will necessarily be more memory-efficient.

> I think this will smack the hands of developers who don’t manage RAM well And hopefully kill Electron. I have never seen the point of spinning up a 300+Mb app just to display something that ought to need only 500Kb to paint onto the screen.

Hopefully just kill off the javascript for everything mindset to be honest.

Re: The RAM shortage could last years

#450

Post by [deleted] »

ethersteeds wrote: post #447
nilkn wrote: post #224

As an aside, recently I wanted to refresh my gaming PC, but the price shock and general lack of availability of buying components individually made it seem hardly worth it, so I just kept deferring the project. Then, mostly by chance, I saw that my local Microcenter had some pre-builts for sale, and I ended up picking one up for The last time I built a gaming PC was upwards of a decade ago, and at that time the preva…

> I ended up picking one up for I'm struggling to put this in context. For comparison, what was your budget for refreshing the pc you had? Were the planned upgrades going to exceed $5k at current prices? Or is the situation that a pre-build machine with far better components was now only marginally more? Or is it that pre-built gaming PCs have stopped being a joke? I had the experience building a bicycle: I was certa…

[deleted]
Post reply on HN