Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High-Bandwidth Flash offers efficient storage for model weights

spectrum.ieee.org

Post reply on HN
Print view View original

11–20 of 27 posts

Re: High-Bandwidth Flash offers efficient storage for model weights

#13

Post by Weryj »

dlcarrier wrote: post #10

Intel really missed out, when they discontinued the Optane line, right before the RAMpocalypse.

Wasn’t quite right before, but Optane was sold off, so it’s sad the buyer just sat on the IP.

Also the selling point was latency, but I suspect we’ll see bandwidth being the important metric with AI using deep pipelines to stream in the weights in a latency insensitive manner.

Re: High-Bandwidth Flash offers efficient storage for model weights

#14

Post by ACCount37 »

That makes some sense. NAND Flash is massively parallel by its nature. That is rarely exposed outside the die though. You'll have that 8 bit double data rate bus and you'll learn to like it.

Now that model inference at scale is a thing though? Model weights, cached prefixes? There's a considerable demand for "slow writes, fast high bandwidth reads" memory. And every bit of storage you didn't have to use RAM for you can use for fatter KV cache instead.

Re: High-Bandwidth Flash offers efficient storage for model weights

#15

Post by ACCount37 »

jmward01 wrote: post #9

I didn't see cost. If cost is similar to flash drives then this could be massive. Every GPU ships with a lot of ram still but for AI inference and games you have 10TB of this stuff to stuff all your textures into and all your model weights in. For fine tuning models this would also be great if using lora or similar.

I expect it to be closer to flash than to RAM, but still expensive. It'll require some special NAND flash dies, and "good" dies too.

Although they'll be able to do the usual thing, and "recycle" the bad dies by selling them off to be used in normal flash drives and SD cards.

Re: High-Bandwidth Flash offers efficient storage for model weights

#16

Post by AlphaSite »

sixdimensional wrote: post #2

So like.. conceptually kind of like memory mapped files on fast flash persistent storage, IIUC? Or maybe it's more like GPU-managed demand paging, caching and DMA? That could get you the capacity and better I/O characteristics. I'm curious about how the unifying architecture is going to evolve between CPU/GPU having direct access to a singular pool of memory/storage also. I also keep wondering when memristor technolo…

AMD had an interesting take on this around a decade ago. Never went anywhere but very cool tech.

Re: High-Bandwidth Flash offers efficient storage for model weights

#17

Post by imtringued »

This feels like an exceptionally bad tradeoff. HBM is expensive due to the packaging cost and yield problems. Now you're trading the relatively inexpensive but high performance DRAM for Flash while splitting the addressable market into training and inference only hardware.

The low capacity of HBM isn't really a mistake. It's a design decision to keep the bandwidth to capacity ratio high. HBM systems with 96GB of memory tend to have around 3.5 TB/s which is a ratio of 35:1, meaning your theoretical maximum is 35 tokens of inference per second assuming you use the full storage just for parameters.

If you massively increase the capacity but keep the bandwidth the same, you just end up lowering this ratio. Your system is overall smaller, but it also has less performance.

This makes High-Bandwidth Flash an extremely niche product or the equivalent of industrially processing lampante olive oil and mixing into high quality olive oil. E.g you're spending an extreme amount of effort on making a worse product that is only marginally cheaper in absolute terms, but more expensive in terms of price to performance ratio.

Re: High-Bandwidth Flash offers efficient storage for model weights

#19

Post by Tade0 »

imtringued wrote: post #17

This feels like an exceptionally bad tradeoff. HBM is expensive due to the packaging cost and yield problems. Now you're trading the relatively inexpensive but high performance DRAM for Flash while splitting the addressable market into training and inference only hardware. The low capacity of HBM isn't really a mistake. It's a design decision to keep the bandwidth to capacity ratio high. HBM systems with 96GB of memo…

They're reporting 1.6TB/s from a 512GB stack - even if they loaded just that 96GB model into it, effectively wasting most of the capacity, it appears they would arrive at a much cheaper system.
Post reply on HN