I have a feeling Intel got rid of Octane a couple years too soon.
Hot Chips 2026: Applying High Bandwidth Flash (HBF)
11–20 of 21 posts
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#12I have a feeling Intel got rid of Octane a couple years too soon.
Yeah, (Optane) 3D XPoint from Micron would have a decent density speed trade-off. I don't know whether it makes the margin they can get from RAM though. Or whether CiM for masked ROM with a 3mo wait for parameter update (and much lower power) is a better trade-off for inference.
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#13So far we have seen a lot of little things built into mainstream chips for the use of hyperscalers, things that don’t require big changes to the software stack. This proposal would require big changes to the software which may mean it remains a proposal.
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#14Earlier quoted context omitted.
Likely much worse write endurance than DRAM, since it’s still flash underneath. The saving grace is that model weights are mostly read-heavy, so endurance may matter less than it sounds.
True, but every time you load a new set of model weights, you are spending writes.
Yeah, but how often is this? Certainly there are weird hobbyist edge cases that don't do well with this, that's true of anything, but a provider is loading weights once every few months
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#15This sounds promising. I am surprised the capacity on the table ends at 512GiB. Is this per chip or per device?
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#16Earlier quoted context omitted.
True, but every time you load a new set of model weights, you are spending writes.
Yeah, but how often is this? Certainly there are weird hobbyist edge cases that don't do well with this, that's true of anything, but a provider is loading weights once every few months
It all depends how many models you are serving from that flash and whether they all fit in there together. If you need to evict and load models, the flash will die a horrible death. This is one resource you should NEVER underprovision.
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#17I had to double check those figures on Sk Hynix office web site [1], and it is not a typo or wrong capital " B ". It really is 3 TB per second. I literally paused for 5 min and thought how is this even possible. [1] https://news.skhynix.com/en/hbf-at-fms-2026/
The crazy part is the interlink: flash bandwidth scales very well with capacity, the challenge is getting that much bandwidth in and out of the chip.
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#18Is this the same idea John Carmack had? ( https://x.com/ID_AA_Carmack/status/2074248758422864226?lang=... ) "Memory cost and capacity are significant issues for AI accelerators. Unlike game rendering, model inference can have a deterministic memory access pattern. You don’t need “random access memory” at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as contin…
Product companies have been talking this up for a year, eg: https://www.sandisk.com/en-ap/company/newsroom/blogs/2025/me...
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#19Earlier quoted context omitted.
Likely much worse write endurance than DRAM, since it’s still flash underneath. The saving grace is that model weights are mostly read-heavy, so endurance may matter less than it sounds.
True, but every time you load a new set of model weights, you are spending writes.
Sure, but 1 write per month is essentially zero even for flash.
Re: Hot Chips 2026: Applying High Bandwidth Flash (HBF)
#20I had to double check those figures on Sk Hynix office web site [1], and it is not a typo or wrong capital " B ". It really is 3 TB per second. I literally paused for 5 min and thought how is this even possible. [1] https://news.skhynix.com/en/hbf-at-fms-2026/
This is almost entirely dominated by the read circuitry and the data path: it’s still taking 1/6 of a second to read the whole chip, which means that the flash cells aren’t working hard at all. (And that pitting the full weights of a dense model on these chips while using anywhere near all the capacity is a nonstarter if you intent to stream the weights as you run inference.)
I think the theory was that when each weight will be needed is predictable, so the latency can be hidden by fetching earlier (or more likely building the data in such a way that streaming it linearly brings the right weight at the right time)