81–90 of 98 posts
Earlier quoted context omitted.
Their GPUs as sold already include RAM.
Yes, but they don't fab their own RAM. It's a cost center for them.
Earlier quoted context omitted.
Optane is crying and punching the walls right now.
Does optane have an advantage over RAM here?
512 GB.
It was slower than conventional DRAM.
But for AI models, Optane may have an advantage: it's bit-addressable.
I'm not aware of any memory controllers that exposed that single-bit granularity; Optane was fighting to create a niche for itself, between DRAM and NAND Flash: pretending to be both, when it was neither.
Bit-level operations, computational units in the same device as massive storage, is an architecture that has yet to be developed.
AI GPUs try to be such an architecture by plopping 16GB of HBM next to a sea of little dot-product engines.
Earlier quoted context omitted.
I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.
I would say that increasing RAM to avoid engineering a solution has long been a successful strategy. i learned my RAM lesson when I bought my first real linux PC. it had 4MB of RAM, which was enough to run X, bash, xterm, and emacs. But once I ran all that and also wanted to compile with g++, it would start swapping, which in the days of slow hard drives, was death to productivity. I spent $200 to double to 8MB, and…
Thank you. Now I feel a log better for dropping $700 on the 32MB of RAM when I built my first rig.
Earlier quoted context omitted.
It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worryin…
I'm just fascinated that the response/demand to running out of RAM is "Just sell us more RAM, god damn!" instead of engineering a solution to make due with what is practically (and realistically) available.
It is possible that compressing and using all of human knowledge takes a lot of memory and in some cases the accuracy is more important than reducing memory usage.
For example [1] shows how Gemma 2B using AVX512 instructions could solve problems it couldn't solve using AVX2 because of rounding issues with the lower-memory instructions. It's likely that most quantization (and other memory reduction schemes) have similar problems.
As we develop more multi-modal models that can do things like understand 3D video in better than real time it's likely memory requirements will increase, not decrease.
Earlier quoted context omitted.
Didn’t nvidia try to block this in software by slowing down mining? Seems like we just need consumer matrix math cards with literally no video out, and then a different set of requirements for those with a video out.
But Nvidia doesn't want to make consumer compute cards because those might steal market share from the datacenter compute cards they are selling at 5x markup.
Nvidia's approach to software certainly deserves scrutiny, but their hardware lineup is so robust that I find it hard to complain. Jetson already exists for low-wattage solutions, and gaming cards can run Nvidia datacenter drivers on headless Linux without issue. The consumer compute cards are already here, you just aren't using them.
Earlier quoted context omitted.
Does optane have an advantage over RAM here?
Optane products were sold as DIMMS with single-DIMM capacity as high as 512 GB. With an Intel memory controller that could make it look like DRAM. 512 GB. It was slower than conventional DRAM. But for AI models, Optane may have an advantage: it's bit-addressable. I'm not aware of any memory controllers that exposed that single-bit granularity; Optane was fighting to create a niche for itself, between DRAM and NAND Fl…
That's an advantage over NAND but not over DRAM. Fundamentally, DRAM is also bit-addressable, but everybody uses DRAM parts with memory cells organized into a hierarchy of groupings for reasons that mostly apply to 3D XPoint memory.
Earlier quoted context omitted.
Yes, but they don't fab their own RAM. It's a cost center for them.
That's not what a cost center is. There is an opportunity for them to make more money by putting more RAM into their GPUs and exposing themselves to a different market. Whether they physically manufacture that RAM doesn't matter in the slightest.
Earlier quoted context omitted.
still wondering why we can't have gpu's with sodimm slots so you can crank the vram
I believe that the issue is that graphic cards require really fast memory. This requires close memory placement (that's why the memory is so close to the core on the board). expandable memory will not be able to provide the required bandwidth here.
Earlier quoted context omitted.
Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards. If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it w…
How comes you don't care about full bandwidth?
If a model takes twice as long to run.... I'll live. Worst-case, it will be mildly annoying.
If I can't run a model, that's a critical failure.
There's a huge step up CPU->GPU which I need, but 3060 versus 4090 isn't a big deal at all. Indeed, the 24GB versus 16GB is a bigger difference than the number of CUDA cores.
Earlier quoted context omitted.
I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.
I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though. While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inferen…
I would gladly buy a card that ran a touch slower but had massive Vram, especially if it was affordable, but I guess that puts me into that camp of enthusiasts you mentioned.